On call, you don't pick the work. Billing. An email pipeline. A service I have never opened. Then billing again. You get to focus on one thing until the next one lands.
Each one needs different context. By the time I've got my head around it, I'm usually on the next one.
But luckily, someone invented agents.
Agents can hold all of it.
What they can't do well yet is tell me if it worked. The agent that did the work is also the one writing the report. How can you trust that?
Last time I wrote about shipping software I can't read. The question that came up since: fine, but how do you know it works?
For a long time my answer was "Agent told me it works & the tests pass." Which is worth almost nothing.
So I stopped checking code and started checking claims. What follows is the on-call version, where being wrong in public costs the most.
The loop
Something happens. An alert fires, or a customer reports a bug, or I want to build a thing. I tell an agent what I want in plain language, usually badly, usually out loud, hands free.
From there it runs:
- It investigates until it can explain the root cause.
- It hands me a plan. Gate one.
- It builds, and other agents try to break what it built.
- It runs the thing for real.
- It hands me the proof. Gate two.
- I say yes, and the pull request opens.
Two gates. Everything between them is the agent's problem. My job is those two pages, and the skill I've written for it is mostly a list of ways an agent can hand me a page that looks like proof and isn't.
The whole thing is a folder. Nothing exotic in it, just instructions the agent follows the same way every time.
on-call-ticket-shepherd/ ├── SKILL.md # the router. picks the mode, loads what it needs ├── references/ │ ├── worker-delivery.md # the six phases, and both gates │ ├── ticket-shepherd-process.md # what goes on each page │ ├── alert-triage.md # when an alert fires at 2am │ └── provider-access.md ├── scripts/ │ ├── render-approval-artifact.mjs # builds gate one │ ├── render-verification-artifact.mjs # builds gate two │ └── artifact-checks.mjs # refuses to build a page that cheats └── templates/ ├── approval-one-pager.html └── verification-one-pager.html
SKILL.md is a router. It decides what mode we're in and loads only the instructions that apply, so the agent never reads the whole folder at once.
Most of the words in there are not about on-call at all. They're about what the agent is allowed to tell me, and what it has to show me before it's allowed to say it worked.
Gate one: the plan has to be readable by me, specifically
Not by an engineer. By me.
I'm a visual thinker, and I don't have twenty years of software vocabulary in my head. If a plan hands me a paragraph with three terms I'd have to look up, I will do the thing everyone does, which is skim it and say "looks good." That is a signature dressed up as a review.
So the plan is an HTML page, and the rules for it are strict:
Sixth to eighth grade language. One idea per sentence. Diagrams flow top to bottom and cap at six nodes, because seven is where I stop reading and start pattern-matching. The first screen has to answer the decision without scrolling.
Two flowcharts side by side: how it works today, how it works after. That pair does more for me than any amount of prose. I can see the user's path change.
The other thing a page does: it survives the switch. I approve a plan on Tuesday, the proof shows up next morning, and by Wednesday I have been in three other parts of the product. I don't have to remember Tuesday.
The page was written for someone with no context, which by then is me. What matters is that the proof is easy to understand once I come back to it.
Then two confidence numbers.
- How sure are we about why this happened?
- How sure are we that this fix does what we say?
A plan can honestly say 45% on the cause and 90% on the fix, and that tells me something real: we're patching something we don't fully understand yet.
And if the root cause isn't established, there is no plan. The agent isn't allowed to build me a beautiful approval page out of a hypothesis. It goes back to investigating and tells me the exact piece of evidence it's missing.
Put together, the page always looks like this:
One more rule that sounds petty. If nothing is built yet, the first line has to say so. Agents write plans in the past tense, so an unbuilt plan reads like a finished job.
The eighty percent I don't watch
After I approve, most of the work happens without me.
The agent writes the code, then other agents attack it from all angles. I run many different models, which really matters. A model checking its own work will often agree with itself.
Every finding ends one of three ways: fixed, refuted with evidence, or waived by me.
Gate two: proof, not a report
This is the page I spend half the time on.
The agent has to prove it works by running it. Describing it does not count. Neither do passing tests.
The status block has three lines, each one pass, not run, or blocked.
Code built. Tests passed. Real flow run. (with proof!)
If a user can see it, run it for real. A screen means screenshots. A flow means a video of the browser. An email means the exact copy, not the fixture with placeholder text, which I have put in front of people.
If the whole flow cannot run locally, run the closest thing and name what you could not cross. That is partial. Partial is fine. Faking complete is not.
The bullshitometer
I didn't plan any of these rules. Each one got added after I fell for something. Three of the twelve, straight out of the file:
## on-call-ticket-shepherd - hard rules (3 of 12)
4. Re-fetch immediately before every external write or status
claim. Bind review evidence to the exact PR head; a new
commit invalidates affected evidence.
9. Adjudicate every high/medium adversarial finding as fixed,
evidence-backed refuted, or explicitly deferred by Marvin.
No unresolved blocking finding crosses a gate.
11. Every fix needs production-shaped local end-to-end proof
when feasible. Unit tests, mocks, lint, and types cannot
by themselves mark the real flow verified.
The other nine are versions of the same problem.
Reading them back, they're all the same lie:
The agent almost never says something false. It says something true about a smaller thing than the thing you asked about.
That's the whole tell.
So the skill defines the words. "Verified" is only allowed when the real flow ran end to end on my machine, with a receipt at the end of it. Tests, mocks, types and lint cannot buy that word.
So the questions I ask at a gate are boring and always the same:
- What exactly did you run, and on which commit?
- What did you see happen at the end, not what did you expect?
- What's the strongest reason this is wrong?
- What did you not check?
- Which of these numbers is a measurement and which one is a vibe?
That last one catches the most. Agents produce confident-sounding numbers with no source, and asking where one came from is free.
The other move is cheaper still. I ask for the same thing twice, in different words, a few minutes apart. A real result is stable. A generated one drifts.
What I'm actually doing when I stare at that page
Not checking correctness. I can't, and pretending otherwise would put me right back where I started.
I'm checking scope. Does the proof cover the claim, or does it cover a smaller thing sitting next to the claim. That's a question you can answer without reading a line of TypeScript, and it's the question that catches real problems.
I'm also checking whether the page is trying to make me feel good. Artifacts that are working read a bit flat. When a page is smooth and every box is green and the tone is a little proud, that's when I slow down. Enthusiasm is a smell.
The scoreboard
Everything above is me checking one page at a time. It says nothing about whether the setup is getting better or worse.
The fix came out of a week on call. Every issue that week had a known answer, so an engineer turned the week into a fixed set of cases and pointed the review agent at all of them.
That is the difference between an opinion and a score. When I change a rule or reword a prompt, I don't guess whether I helped. I run the set and see what it missed.
Prompts rot. You tune them for the last thing that annoyed you and find out months later. This is the only part of the setup whose answer key came first.
Someone already shipped this
A talk by Lauren Tan, an engineer at Cursor, takes the same idea further. Verification is what turns an agent you supervise into one you can hand work to.
The difference is that this one ships. It's a Cursor plugin called pstack, and one command, create-verification-skill, reads your repo and writes a verification skill for that project. Mine took months of being fooled. The same plugin has an eval playbook.
The part I can't fix
The page is written by the thing I'm checking.
I've made that better in the obvious ways: different models, adversarial passes, rules that force specific claims. "It works" is unkillable. "I triggered it at 14:02, the job ran, and the row appeared in the table" either happened or it didn't.
I'm not going to pretend the loop closes. A determined bad artifact would get past me. It hasn't happened much because the failure mode isn't malice. Agents are agreeable. They want to hand me a green page, and my setup is built against that.
The wrong skill
I spent my first months on the team thinking the skill I was missing was reading code.
The skill I was missing was reading claims. Knowing what a word like "verified" is doing in a sentence. Noticing when a proof is one size smaller than the thing it proves. Asking what got left out.
That one is portable. I learned most of it before I joined an engineering team, watching people present numbers that were technically accurate.
When most of a team's code is written by agents, nobody is reading all of it. Everybody is reading claims about it.
Two gates and a working bullshitometer. That's what I've got. It's less than knowing, and a lot more than trusting.