Between August 2 and October 7, my three custom review agents ran eight times in total. The two general-purpose agents that ship with the tool ran 63 times in the same window.
Nothing was broken. The agents were configured, briefed and one command away. A rule I wrote told the model not to call them, and it did exactly what I told it to.
What I built
I work with Claude Code most days. Over the summer I set up three custom agents, each a separate model session with its own instructions, its own tools and a narrow job:
- A reviewer that reads a finished change against the task it was meant to do, and challenges it.
- A verifier that gets the acceptance criteria and the finished work, never my argument for why it’s done, and returns pass or fail per criterion.
- A scout that chased one question through the code so the research didn’t flood my main session.
All three are read-only. None of them can edit a file. That’s the point: a checker that can fix what it finds stops being a checker.
The rule that benched them
In July I ran a small coding benchmark: three tasks built to fail without throwing an error, four models, each working alone. The strongest model passed all three, and the passes held with my hooks switched off. So on August 2 I wrote a per-model rule, injected on every prompt: don’t delegate verification, don’t spawn a reviewer, verify in your own loop.
The measurement was real. The rule went further than the measurement did.
The bench was three code tasks. Most of my work isn’t code tasks. It’s prototypes, decision documents, specs a developer has to build from, and data stories where the claim is the deliverable. The bench never measured any of that, and the rule covered all of it.
The model followed the rule on roughly 84% of my prompts, the share that ran on that model. Exploration agents kept running, because the rule allowed those. The checkers went quiet. Workflows I launched for big audits kept spawning their own skeptics; it was the everyday moments, like calling something done or handing off a spec, that lost their second look.
What the audit found
In October I counted agent use from my session logs instead of remembering it. Two numbers stood next to each other:
- During the rule: reviewer 1, verifier 4, scout 3. Against 34 and 29 for the built-in exploration and general-purpose agents.
- When the custom agents did run on real work: 7 of 7 changed what happened next. A fix got rescoped, a claim got pulled, a doc got corrected before someone else read it.
Rarely used and always useful is not a tool problem. It’s a routing problem.
What changed
On October 7 I replaced the ban with five triggers, each tied to a moment in the work rather than to the model:
- Wide reading across many files, where only the conclusion matters → an exploration agent.
- A decision or a vocabulary word changed, and several documents must now agree → a fresh-context doc audit.
- A spec or handoff about to go to a developer → a cold read as that developer.
- A change about to go to its reviewers → the cold reviewer.
- About to say “done” on multi-step work → the verifier, given the criteria and never my reasoning.
The scout was parked, because exploration agents already do its job. Two new agents were built for triggers 2 and 3. One checker per deliverable, never one per finding.
The next day I found a second, quieter version of the bug. A code-edit gate exempted the benchmarked model by matching part of its name. A newer model whose name contained the same string inherited the exemption without ever being measured. Now it matches exact model names only. A new model is gated until it’s re-benched.
In the two days since: the doc auditor ran 23 times, the developer cold read 6, the verifier 6, the reviewer 2.
Where I’d push back on the common advice
The usual line on agents is “use them freely, fresh eyes beat self-critique.” My own bench disagreed on code tasks, and I recorded that disagreement on purpose. I still think it’s right for code a strong model can check itself.
Where I’d push back is on treating either answer as a setting. “Always delegate” and “never delegate” both skip the question that matters: what kind of work is this, and what would prove it wrong? An agent earns its place at the moment where my own reasoning is the thing most likely to be mistaken. That’s a property of the moment, not of the model.
Where this goes sideways
Two days is a burst, not a habit. The post-fix numbers are what a new tool looks like the week it’s new. The honest test is a month from now, with each run logged: what it found, and whether it changed the outcome.
“Changed the outcome” is my own judgment of my own runs. Seven is a small number, and I’m the one who scored it. It’s enough to say the agents weren’t useless. It isn’t enough to say how useful.
The fix for too few isn’t too many. In late September one run of mine spawned a verifier per finding and ended at 770 agents. That’s the opposite failure, and it’s why “one checker per deliverable” is now written down.
The bench still says what it said. On code tasks, the strong model passed without a review layer. I haven’t re-run it on the work the rule actually covered: documents, prototypes, claims. That run is next.
Monday version
Count your agent use from the logs, not from memory. Deduplicate by call ID, since one message can be written as several lines. Then put each agent’s count next to one question: the last time it ran on real work, did anything change?
If the answer is “rarely used, always useful,” don’t tune the agent. Find the rule that’s routing around it.
Thanks for the read.
Provenance
- Agent counts: my own Claude Code session transcripts, counted 2026-10-09 at 15:18 ET. Each agent call is counted once by its tool-call ID. Top-level sessions only; agents spawned inside a workflow aren’t in these numbers. Sessions run inside my benchmark folders are excluded from the period counts. Ban window: 2026-08-02 through 2026-10-07. “Two days since” is 2026-10-08 and 2026-10-09.
- An earlier count was wrong. The one-line grep I’d written down for this undercounted the general-purpose agent (31 against 38 across all time) because its pattern stopped at the first closing brace inside a prompt. The numbers above come from parsing each transcript line as JSON instead.
- “7 of 7”: from my October audit of the custom agents’ real-work runs, each judged by whether the next step changed. Not an independent score.
- 84%: the share of my prompts that ran on the model the rule targeted, from the same audit.
- The benchmark: three silent-failure coding tasks with answer keys, four models working solo, synthesized 2026-07-27, plus an ablation that removed my hooks. Personal-scale; not a published evaluation.
- What this is evidence of: agent configuration I built and the usage I measured on myself. It isn’t evidence of team adoption. That’s a separate number I don’t have yet.
