When the Tripwire Cries Wolf: Small-Sample Review Metrics in an Agent Fleet

The night two model tiers "degraded" at once

At 02:33 UTC on September 1 my fleet's approval bot posted a red alert: the fable tier's reviewer pass-rate had fallen to 58% over the last seven days against an 82% baseline, a 24 point drop. Two hours later the same alert fired for opus5: 54% against 76%, down 23 points. Both tiers were pulled off automatic routing for 120 minutes, and every new task went to a neighbouring tier on the ladder.

Two independent model tiers failing in the same night looks like a real event. The fleet had shipped a config change the previous day, the reviewer prompts had been touched the week before, and the natural reading was that something upstream had broken code quality for everyone. I spent the next hour pulling the numbers, and the conclusion was the opposite: nothing had degraded. The tripwire had measured a hard week and reported it as two broken models.

This article is the post-mortem. It is about a specific failure mode of automated quality metrics in agent pipelines, correlated small samples, and about the specific fix that now stops it. If you run any kind of pass-rate alarm over LLM output, the numbers below are probably closer to your situation than you would like.

How the pipeline and the tripwire work

My agent-corp setup is deliberately narrow. A daemon on a dedicated server polls a GitHub Projects board, claims a task, and runs it as headless Claude Code inside a Docker container. The executor writes the code and opens a pull request. A second Claude session, started fresh with no memory of the executor, reviews the PR sceptically and sends it back for revision until it passes or runs out of rounds. An optional cross-vendor skeptic, an OpenAI model run through the Codex CLI (gpt-5.3-codex), then tries to refute the PR. It never writes code; it only argues. The final merge is a Telegram button that I press.

Tasks are routed to model tiers: opus5, fable, sonnet5, grok. The daemon picks a tier automatically unless the board pins one. Because tiers can be swapped and vendors can ship silent regressions, I built a model-health tripwire on top of the review log. Every reviewer verdict lands in a JSONL file with the tier, the PR, and the outcome. The tripwire compares each tier's pass-rate over the last seven days with its own thirty-day baseline. If the recent rate is at least 20 points below baseline and there are at least 10 recent verdicts, the tier is marked as degraded, an alert goes to Telegram, and the router de-prefers that tier for two hours, extending the window while the condition holds.

That design has a sensible shape: per-tier baseline, a minimum sample, an automatic but reversible action. It also has a hole big enough for a whole week of work to fall through.

What the numbers actually said

The two alerts, written out as fractions instead of percentages, were:

A two-proportion z-test on those pairs gives p≈0.13 for fable and p≈0.12 for opus5. Neither is close to a conventional significance level. The 95% confidence interval on a rate estimated from 12 or 13 observations is roughly plus or minus 25 percentage points, which is wider than the 20 point threshold the alarm is watching for. In plain terms, the alarm was set to fire on a movement smaller than the measurement error of its own input.

A minimum sample of 10 sounds like a guard, but it is not a guard against this. With a 20 point threshold and n around 12, a tier whose true quality never changes will trip the alarm on random variation roughly once in every eight evaluation windows. Run four tiers and evaluate daily and you get a false alarm every couple of days from noise alone. That is the arithmetic behind the alerts, and it is enough on its own to explain the night.

Why both tiers moved together

The z-test says each alert was weak evidence. The second finding explains why two weak alerts arrived together, which is the part that made them look convincing.

I pulled the recent pass-rate for every tier, not just the two that had tripped. sonnet5 was at 66%. grok was at 53%, which its alarm read as an improvement because its baseline was a weak 44%. Every tier had landed in the 53 to 66% band in the same week. When all of your models move together, the parsimonious explanation is not that all of your vendors regressed on the same Tuesday. It is that the workload changed.

It had. The week's queue was dominated by daemon, CI and deploy-path work: a paths-ignore bug that silently stopped prompt-only merges from deploying, a run of deep infrastructure PRs, several tasks that went through three or four revise rounds before passing. Reviewer pass-rate is not a property of the model; it is a property of the model applied to a particular week's tasks. The metric measured the epoch and attributed it to the tier.

There was a third effect that turned one hard week into two alerts. The review stage fans out: the same PR can be reviewed under several tiers within minutes, and each verdict is logged under the tier that produced it. One difficult PR therefore appears in the recent window of opus5 and in the recent window of fable, and its failures count against both. The two "independent" per-tier samples were largely the same PRs counted twice. The samples were correlated by construction, and a threshold that assumes independent observations does not know that.

So the picture was: small samples, a threshold inside the noise band, a workload shift that dragged every tier down, and a logging scheme that double-counts the same hard PRs across tiers. None of those is a model regression. Together they produce a very convincing imitation of one.

The fix, in three parts

I filed the finding as a task on the board and let the fleet handle it like any other task. The router sent it to the grok tier, one of the tiers that had not tripped, and the fix was reviewed and merged the same morning through the usual chain: fresh reviewer session, tests, my button in Telegram. The change is small and it touches only the tripwire.

Raise the minimum sample from 10 to 20. At n=20 the confidence interval on a rate is about plus or minus 20 points, which is the lower edge of where a 20 point threshold has any power. It is still not a lot of data, but it is no longer smaller than its own error bar. A lower minimum would just reintroduce the once-per-eight-windows false alarm.

Add a significance gate. Before alerting or touching routing, the tripwire now runs a pooled two-proportion z-test on recent versus baseline and requires a two-sided p below 0.10. The choice of two-sided rather than one-sided was deliberate and is recorded in the PR: a one-sided test at 0.10 would still have fired on the September 1 data, so it would not have fixed the incident that motivated the change. The gate can be disabled by setting the maximum p-value to zero, which leaves the routing kill-switch as a separate, independent escape hatch.

Put the evidence in the alert. The Telegram message now carries the sample size for both windows, the p-value with a note that it is a two-sided z-test, and a Wilson 95% interval on the recent rate. A line that says "7 of 13 versus 26 of 34, p≈0.12" is read very differently from a line that says "down 23 points". The old text was scarier than the data, and it had me reaching for the wrong explanation for an hour.

Two unit tests pin the behaviour. A fixture replaying the September 1 double-trip, reconstructed in the exact shape of the review log, must not fire under the new defaults; on the old code it fails with both tiers marked degraded, on the new code it passes. A second fixture with a sustained real drop, 76% to 50% at n of at least 30, must still fire. Both are in the repository's test suite alongside sixty other review-metric tests.

What the fix does not do is de-correlate the samples. Aggregating per PR first, so that one pull request is one observation attributed to the tiers that reviewed it only as a tiebreak, is the right long-term shape, and it is listed as a follow-up. The minimum sample and the significance gate were enough to satisfy the acceptance tests, and the fleet stayed inside that scope rather than expanding the change on its own.

What I would tell another team

The specific lesson is about pass-rate tripwires, but the general one applies to any metric that a team lets take automatic action.

Compare your threshold to your error bar before you ship the alarm. If the smallest change you want to detect is narrower than the confidence interval at your minimum sample, the alarm is a random number generator with a Telegram integration. Work out the interval at your minimum n once and write it next to the threshold in the config file.

When everything moves together, suspect the input, not the models. A regression in one vendor's model shows up in one tier. A hard week shows up in all of them. Plot all tiers on the same chart before believing any one alert.

Know what your unit of observation is. In a fan-out review stage the observation is the PR, not the verdict. If the same artefact is logged under several labels, per-label samples are not independent and per-label significance tests overstate the evidence.

Separate monitoring from decisioning. The dashboard can be sensitive; it costs nothing when it wobbles. The thing that removes a tier from routing must be conservative, because the false-positive cost is real: for two hours my fleet ran on tiers it would not otherwise have chosen, and I spent an hour of founder time on a regression that did not exist.

Make the alert carry its own uncertainty. Every alarm my fleet raises now shows the fraction, the sample size and the interval. Percentages without denominators are how a 7 out of 12 turns into a crisis.

FAQ

How do you tell a noisy epoch from a real model regression?

Look at three things together: whether the per-tier drop clears a two-proportion significance test at your chosen level, whether other tiers moved in the same direction over the same window, and whether the recent samples share the same underlying tasks. A real regression is usually significant, isolated to one tier, and persists as the window fills with new, unrelated work. A noisy epoch fails at least one of those checks.

Why not simply raise the drop threshold instead of adding a p-value gate?

A fixed threshold ignores sample size. A 20 point drop is meaningless at n=12 and alarming at n=60. A significance test scales with n automatically, so the same config behaves correctly for a quiet week and a busy one. The minimum sample is kept as a floor so that the test itself is not run on absurdly small windows.

Does the tripwire still catch a genuine sustained drop?

Yes. The regression test for the fix replays a sustained fall from 76% to 50% at 30 or more recent reviews and requires the alarm to fire. That case has p well under 0.10 and clears the minimum sample, so both gates pass. What no longer fires is a 20 point swing on a dozen correlated verdicts.

Why let the fleet fix its own metrics instead of doing it by hand?

Because the fix is a normal code change with clear acceptance criteria, and that is exactly what the pipeline is for. The task had a fixture to reproduce, two behaviours to pin with tests, and a config default to document. The executor wrote it, the fresh reviewer session checked it, the tests ran, and I read the PR and pressed merge. Doing it by hand would have taken me about as long as writing the task description did, and the task description is the part that carries the reasoning.

Related reading