# Unknown Is Not Clean: Metrics for Code Your Agents Write

Author: Maksym Tytarenko | Date: 2026-08-20 | Category: AI & ML | Tags: ai-agents, engineering-metrics, observability, developer-productivity, code-review
Canonical: https://www.tytarenkoagency.com/blog/unknown-is-not-clean-metrics-for-code-your-agents-write

> Ninety-nine percent of what a coding-agent fleet burns is re-reading, not writing. Here is the metric set I built to see what that spend actually buys, and the five rules it taught me, starting with the one that made my first clean-looking report a lie.

# Unknown Is Not Clean: Metrics for Code Your Agents Write

My fleet of coding agents ships pull requests around the clock. For months I could tell you, to the cent, what any of it cost. I could not tell you what the money bought.

That asymmetry is easy to fall into. Cost and duration arrive for free. Every container run already writes a row with a price and a stopwatch, every model call is a metric with a label. Quality does not arrive at all. So you end up with a dashboard that answers "how much" in four decimal places and answers "was it any good" with a shrug, and you optimise the half you can see.

I spent an evening fixing that. The interesting part was not the dashboard. It was how many of my first numbers were wrong, and that every one of them was wrong in the direction that flattered me.

## The measurement that reframed the problem

Before designing anything I pulled twelve hours of raw telemetry. The token split:

- cache reads: 317,569,508
- cache creation: 3,582,800
- output: 1,161,428
- input: 120,634

![An infographic showing the breakdown of token metrics.](https://media.tytarenkoagency.com/ai-generated/uploads/20260820_032424_95f695f8.png)


Ninety-nine percent of what an agent loop burns is **re-reading**. Output (the actual code, the actual review, the actual decision) is a rounding error next to it.

That single ratio changes what "expensive" means. The bill is not proportional to how much your agents write. It is proportional to **context multiplied by turns**. A long round on a big branch does not cost twice a short one; it costs many times more, because every turn re-reads everything. Twelve hours came to $240. Half of that was not the fleet at all. It was the orchestrating session, running on the most expensive tier available, doing work like reading diffs and locating functions. Spend on the cheapest tier in my routing ladder that day: zero dollars. The ladder existed. Nothing used it.

Useful, but still only the "how much" half. What I could not answer was whether any of that spend had prevented a single defect.

## Five rules for measuring agent-written work

I built the aggregator on artifacts already being written, so a baseline could exist before instrumenting anything new. The rules below are what the build taught me, several of them the hard way, by shipping the opposite first.

![A flowchart summarizing the five rules for measuring work.](https://media.tytarenkoagency.com/ai-generated/uploads/20260820_032335_14b0fba8.png)


### 1. Tripwires are counts, and they print above the money

The quality floor is four numbers: merges where a human overrode a blocking gate, merges whose head carried no bound review verdict, merges shipped with suppression flags, and danger-class changes merged with no external pass.

They are counts, never rates. One merge with no bound verdict is a thing to explain, not a percentage to feel good about. And they render *above* the cost lines, so a cheap week with a tripwire cannot read as a good week. This ordering is the entire point: if you show savings next to nothing else, cost work will quietly buy itself with quality, and the report will congratulate you while it happens.

### 2. Unknown is never clean

My first baseline printed four zeros on those tripwires. It looked like a flawless week.

It was not a week at all. It was a report whose caller had populated exactly one of the four fields. Three guards treated a *missing* field as a passing check. A tripwire that reads zero because nobody looked is worse than no tripwire, because it manufactures confidence.

Now every guard carries its own "unknown" counter, and when any of them is non-zero the report says, in as many words, that the week is not clean but unmeasured. On the corrected run all four guards were unknown across every merge. That is a worse-looking report and a far more honest one.

The same principle bit twice more. The kill counter, which tracks runs that hit their time limit rather than finishing, reads as a floor and never as a count, because a run killed at the wall may write no row at all, and one whole model tier reports a duration of zero. On the day I measured, duration was known for 61 of 71 runs. Print the denominator or the zeros lie.

### 3. Every proxy names the direction it errs in

Nothing here is measured directly. Every number is derived from an artifact written for another purpose, which makes each one a proxy. A proxy whose limits are undocumented becomes a fact by repetition.

So each one states its bias in its own docstring. "Empty rounds" can under-count and can never accuse a round that actually pushed. "Re-read cost" is the money spent re-reading, not the money wasted, because it cannot distinguish a re-read forced by a moved base from one earned by real new work. The escape metric is a trend line, not a verdict on any single change.

Where a proxy can only fail in one direction, say which. Where it can fail in both, say that too. Then a reviewer can catch you when the code disagrees with the docstring, which is exactly what happened to me.

### 4. A quoted number must be re-runnable by whoever doubts it

My first baseline came from a throwaway script that was not in the diff. The numbers were quotable and unverifiable, which is the worst combination a metric can have. It now ships as a command in the repo.

### 5. The denominator is part of the number

My headline figure divided spend booked to merged work by the number of merges. Work that did not merge was still paid for, so the number read about forty percent low. It now divides *all* spend in the window by the merges in it, with the flattering version printed beside it and labelled as the flattering version.

## What the corrections did

Same twelve to twenty-four hours of data, before and after the review passes:

![A table graphic comparing cost metrics before and after corrections.](https://media.tytarenkoagency.com/ai-generated/uploads/20260820_032246_004c491f.png)


| | as first published | corrected |
|---|---|---|
| cost per merged PR | $6.35 | $15.81 |
| re-read bucket | $44.32 (24% of spend) | $83.27 (about half) |
| lead time, 90th percentile | 9.7h | 6.2h |
| quality tripwires | "all four zero" | unmeasured, all four |

Every error but one ran in my favour. The percentile was biased high by a rounding bug that made "90th percentile" return the single worst outlier whenever the sample was small.

The re-read bucket doubling is the one that matters. It doubled because I had been counting repeat reads only *within* a single gate: a change read by four different gates (plain review, cross-vendor skeptic, full review, security) cost nothing at all in my original arithmetic. Four full reads of one diff, priced at zero.

## What the numbers then made me change

**Invalidate by region, not by filename.** When a change merges, every sibling branch sharing a file with it had its review verdict voided. In a repo where the main daemon module runs about eight thousand lines and the config module appears in nearly every diff, "shares a file" is true almost always. So almost every merge invalidated almost every sibling, and the next review re-read a whole branch to re-earn a verdict for code nobody had touched. One branch paid three base merges out of nine commits, five full reviews and two security gates across eight hours, largely to that rule.

The fix narrows it to "did the two changes touch the same *region*". But the honest version of that fix needs one more thing, and this is where measuring saved me: line numbers from two branches are only comparable if both diffs are expressed against the same base. I measured the drift in that daemon module: between one hundred and four hundred eighty lines a day. A single anchor function moved ninety-nine lines in twenty-four hours. My original tolerance was twenty-five lines, which means two changes editing *the exact same function* would have been declared unrelated after a few hours of drift. Comparability now has to be established, not assumed.

**Scope a round, do not shorten it.** Two agent rounds that evening hit their time ceiling and pushed nothing at all, burning seventy-five minutes and their full context cost to return zero. Given cost is context times turns, the remedy is not a bigger limit; it is one finding per round with an instruction to commit before starting the next.

**Route the reading to the cheap tier.** Half the bill was the orchestrator, and a large share of its tokens went on reconnaissance: reading diffs, finding where things live. That is work for the cheapest model in the ladder, not the most expensive.

## The part that earns the rest

Both of the changes above shipped with a confident safety claim in the commit message. An adversarial review refuted both.

The first claimed it could only ever remove a false invalidation, never create a blind spot. In fact the region check sat *before* the automatic base refresh, so a branch declared unrelated never got refreshed, leaving green-but-stale test results mergeable, with no branch protection behind them. Worse, it was self-reinforcing: the refresh it skipped was precisely what kept branches aligned, so every skip widened the drift that makes the next region answer wrong.

The second claimed every proxy named the direction it erred in and that the code matched. Six findings said otherwise, all confirmed by execution.

Then the part I keep thinking about. I fixed that second one, making "missing data is unknown, not zero" true throughout the pure logic, and twenty minutes later reimplemented the identical mistake one layer up, in the input code I wrote *in the same commit*. A failed API call silently became "zero commits", which turned every agent round on that branch into a false accusation of doing nothing. My own comment above the code asserted the opposite direction.

Fixing a principle in one place does not install it in your hands.

And one bug no unit test could have caught was caught by the first real run: review comments live on the pull request, not on the issue it closes. My lookup queried the wrong one and returned empty every time, so the metric would have read "unknown" forever while looking perfectly healthy. After the fix it reported two empty rounds, which is exactly the number I had watched die that evening.

## What I would tell another founder

Measure quality first, or your cost work will buy itself with quality and the report will not tell you.

Make every saving publish its tripwires alongside. Make missing data say "unknown" out loud, in the render, not just in a field nobody reads. Give every derived number a stated direction of error, then have someone hostile check the code against that statement. Ship the command that produces the number, not the number.

And do not review your own work. I wrote two changes, believed both were safe, wrote that belief into the commit message twice, and was wrong twice. The external pass is not ceremony. It is the only thing standing between a confident sentence and a false one.

## Related reading

- [Why I Rejected an 817-Skill Security Pack and Wrote a 2-File Threat-Model Skill](https://www.tytarenkoagency.com/blog/why-i-rejected-an-817-skill-security-pack-and-wrote-a-2-file-threat-model-skill)
- [I Turned a Telegram Group Into a Software Project](https://www.tytarenkoagency.com/blog/i-turned-a-telegram-group-into-a-software-project)
- [Graph engineering is easy. Not blowing your budget on one workflow is the hard part](https://www.tytarenkoagency.com/blog/graph-engineering-is-easy-not-blowing-your-budget-on-one-workflow-is-the-hard-part)