0.25: The Number That Decides Whether My Agents Forget a Lesson

My agent fleet keeps a short memory per repository: a file of lessons distilled from code review findings, injected into every future task in that repo. This week I found that a fifth of one of those files was the same lesson written seven times, and that the duplicates were quietly pushing distinct lessons out. The fix is one number, 0.25, and most of this article is about why that number and not another one.

How the lessons file works

agent-corp is my autonomous agent fleet. A daemon polls a GitHub Projects board and runs each task as headless Claude Code inside a Docker container. A fresh Claude session reviews the pull request, and when that reviewer asks for changes, a Sonnet call distills its findings into one or two terse lines. Those lines go into .agent-corp/lessons.md in the target repository.

Each bullet is a single sentence of guidance plus a provenance trailer: the pull request it came from, the date, and the verdict that produced it. Three channels write bullets: reviewer findings, the CI fix path, and my own revise notes, which I can keep as a lesson with one press of a Telegram button.

The file is bounded on purpose. It holds at most 40 bullets, and the executor receives the newest 20 as a "Known pitfalls in this repo" block. My own interactive sessions get a similar block. A bullet can also be retired without being deleted: it is marked overridden when I merge over the finding that created it, stale when the file it names no longer exists, and expired after 90 days. Retired bullets stay in the file for audit and drop out of what agents see.

The 90 day expiry came with a rule I had written down earlier: a pitfall that is still being hit should be re-earned by a fresh finding, which means a new dated bullet. That rule is where the trouble started.

Seven copies of one lesson

Re-earning by appending sounds harmless. In practice the write path was a blind append followed by a first-in, first-out trim to 40 lines. There was no duplicate check anywhere on it.

The clearest damage was in the store for my Claude Code setup repository. Seven of its 40 active bullets told the agent the same thing: bump a minimum test case counter in the test runner script after adding tests. Four of the newest 15, the window that reaches every interactive session, were that one lesson. Every copy also evicted a different lesson from the old end of the file, because the cap counted lines, not ideas.

The same file showed a second symptom. One bullet began with "Correction to the bullet above" and refuted a claim about what an OS detection function returns. The bullet it corrected carried no retirement flag. For as long as both sat inside the window, every session received a lesson and its own refutation, side by side.

The site you are reading has the same problem. Its lessons were injected into the session where I am writing this, and two of them are one lesson in different words:

Never approve a PR while any CI check (...) is still pending: wait for all jobs to conclude before verdict.
Never request approval or treat a PR as mergeable while any CI check (...) is still pending: local green runs are not a substitute.

Both came from the same review. They are a fair sample of how duplicates look in these files: never word for word, always the same instruction with a different tail.

The shape of the fix

The change keeps the re-earn rule and changes how it is applied. When a new bullet is a near duplicate of an active one, the writer does not append it. It rewrites the existing bullet's trailer with the new pull request and date, and leaves the text, the position and the original verdict alone. The pitfall is still marked as recently seen, and it still occupies exactly one slot.

def collapse_target(text, bullets, verdict, floor=0.25):
    best = None
    if len(tokens(lesson_text(text))) < 8:
        return None
    for i, bullet in enumerate(bullets):
        meta = parse_trailer(bullet)
        if meta is None or meta["overridden"] or meta["stale"] or meta["expired"]:
            continue  # retired bullets are never refreshed
        if (meta["verdict"] == FOUNDER_NOTE) != (verdict == FOUNDER_NOTE):
            continue  # founder notes and reviewer bullets never merge
        if len(tokens(lesson_text(bullet))) < 8:
            continue
        score = lesson_similarity(text, bullet)
        if score >= floor and (best is None or score > best[0]):
            best = (score, i)
    return None if best is None else best[1]

The comparison runs against the working list before the trim, so two near copies arriving in one call also land on a single bullet. If every new bullet collapses onto a trailer that already cites the same pull request and date, nothing is written at all.

The interesting part is not this function. It is lesson_similarity and the floor.

Why bigrams failed

I already had a near duplicate detector in the fleet. It was built for a different job, spotting repeated candidates in a chat intelligence feed, and it compares texts by Jaccard overlap of word bigrams: break each text into consecutive word pairs, then divide the shared pairs by all pairs.

On lessons, bigrams barely work. A lesson is one sentence of about 20 tokens. Two people describing the same pitfall share the key words but rarely the same word order, so their pair sets hardly overlap.

The two pending CI bullets above make the point. On word bigrams they score 0.158. On single words they score 0.286. Across the whole corpus, the highest bigram score for any pair was 0.22, and the known seven copy class topped out at 0.20, inside the range where distinct lessons also sit. A threshold on bigrams would either collapse nothing or collapse real lessons.

So the lessons path uses single words, while the original detector keeps its bigrams for the job it was built for.

Picking 0.25 from data, not taste

The fleet writes lessons for eleven repositories. Before choosing a floor, the task first produced a report script that takes lesson files as plain paths and scores every pair of active bullets. On a snapshot of all eleven stores taken on November 2, that meant 223 active bullets and 3,211 pairs.

What the table showed:

Every one of the 28 pairs at or above 0.25 was the same pitfall reworded. Same-class pairs ranged from 0.25 to 0.50. The highest scoring pair of genuinely different lessons was 0.2143.

The runner-up distinct pair is the reason the floor is not lower. In the three-waters repository, the 3D fishing game my fleet built, there are two bullets about the same screenshot command. One says to run npm run shots for art-gated pull requests. The other says never to retry npm run shots inside the task container, because the browser download it depends on is blocked there. They share most of their words and say opposite things. They score 0.2121. Merging them would replace a working instruction with its own prohibition, or the reverse.

A similar near miss sits in the store for the Little Prince tale: "measure every painted overlay anchor" against "re-measure one specific anchor set after repainting", at 0.214. Related, but not the same lesson.

At 0.25, the floor sits just above every distinct pair and at or below every duplicate pair in the corpus. Applied to the snapshot, 32 of 223 active bullets would have collapsed, 14 percent of the fleet's lesson memory. The share was uneven: 32 percent in this site's own repository, 25 percent in the Claude Code setup repository, and zero in five smaller stores.

I want to be honest about the margin. The gap between 0.2143 and 0.25 is not wide, and it was measured on one snapshot. The bias is deliberate, though. A missed collapse costs one slot in a 40 line file. A false collapse loses a lesson, and on the evidence of the screenshot pair it can invert one. When in doubt the floor errs toward keeping two bullets.

The token floor

Unigram Jaccard has a failure mode of its own: with very little text, the score means nothing. Two lines of two tokens each, such as "new one" and "new two", overlap at 0.33 while saying different things.

So neither side is compared unless it carries at least 8 content tokens. The smallest real bullet in the corpus has 11, and the median is 17, so the guard never touches a real lesson. It exists to stop short, malformed text from scoring its way into a merge.

Two bullets that are never refreshed

A retired bullet is never a collapse target. If I merged over a finding and its bullet was marked overridden, or the 90 day expiry put it down, a fresh finding of the same class has to come back as a new active bullet. Refreshing the old trailer instead would quietly resurrect a lesson that a merge, a sweep or the expiry had already retired, and the reason it was retired would be lost.

The founder note channel is also kept apart, in both directions. A reviewer bullet must not absorb my own words, and a note I kept must not be replaced by a reviewer's restatement of it. The same exemption already applied when bullets are marked overridden: my merge overrides the reviewer's caution, not my own correction.

Measuring it over time instead of once

Every write now returns how many bullets were appended and how many were refreshed. The refresh count appears in the log line and in the commit subject, and each write records a stats row with that count and the store's active size after the write. A small reader computes duplicate share per store from those rows. It is not wired into my weekly report yet, but the data now accumulates with every write, so the next time I want to know whether 0.25 still separates the corpus, I can read a trend instead of measuring another snapshot by hand.

That matters because the corpus will change. New repositories bring new phrasing habits, and the summarizer's style can drift with a model change. A threshold that was clean on November 2 is a claim with a date on it, not a constant of nature.

What this does not fix

The change consolidates on the write side. It does not resolve contradictions. The bullet that said "Correction to the bullet above" would not be caught by this logic, because a correction and its target can be phrased very differently, and collapsing them would keep the wrong text. Retiring a refuted lesson still depends on the existing paths: an override at merge time, a stale file reference, or the expiry.

It also does not touch bullets that were duplicated before the change shipped. Those age out through the normal expiry or get refreshed when the same finding comes back.

Takeaways

A capped memory needs deduplication at write time. A first-in, first-out cap over lines lets one recurring problem evict distinct ones, and the agent sees less variety exactly where it matters.

Re-earning a lesson should update its evidence, not add a copy. Moving the date and source in place keeps recency without spending a slot.

Check that your similarity measure fits the text length. Bigrams suited longer feed items and failed on 20 token sentences. Single words separated the same corpus cleanly.

Pick the threshold from your own pairs. Score every pair, find the highest distinct pair and the lowest duplicate pair, and put the floor between them with a conservative bias.

Keep retired items and human notes out of automatic merges. A refresh that revives a retired lesson or overwrites a human correction is worse than a duplicate.

Record the metric on every write. A threshold measured once is a dated claim; a stats row per write keeps it checkable.

FAQ

Why not just raise the cap above 40 bullets?

A larger cap moves the problem without solving it. Duplicates would still crowd the injected window of the newest 20 bullets, and every extra line costs context on every task in that repository.

Why word unigrams instead of embeddings?

Unigram Jaccard is deterministic, cheap, needs no model call on the write path, and is easy to audit: anyone can recompute a score by hand. On this corpus it separated duplicates from distinct lessons with a clean gap, so a heavier method had nothing to add yet.

What happens when a duplicate arrives for a retired lesson?

It is written as a new active bullet. Retired bullets are never refreshed, so a lesson that was overridden, went stale or expired has to be earned again by a new finding rather than revived silently.

How did you choose 0.25?

By scoring all 3,211 pairs of active bullets across eleven lesson files. All 28 pairs at or above 0.25 were the same lesson reworded, and the highest pair of distinct lessons scored 0.2143. The floor sits between the two, biased toward keeping lessons apart.

Can two opposite lessons still be merged?

The closest real case, two instructions about the same screenshot command where one says run it and the other says never retry it, scores 0.2121 and stays below the floor. The margin is narrow, which is why every write now records its collapse count, so drift in the corpus can be read from the data rather than discovered by accident.

Related reading

When the Tripwire Cries Wolf: Small-Sample Review Metrics in an Agent Fleet

Four Merge Presses, One Merge: Fixing the Approval Layer of My AI Agent Fleet

PHP2Shell Week: I Ran My Agents' Security Audit on My Own Site and It Found a Leak