Executive Summary

My autonomous AI agent fleet hit two incidents in the same week of July 2026. First, an orphan-reclaim mechanism re-queued two tasks whose agent pull requests (PRs) I had already merged by hand, and the fleet launched fresh executor containers that started redoing shipped work, one of them running Opus at high reasoning effort. Real token burn for zero new value. Second, a Codex CLI upgrade (0.142.5 to 0.144.1) silently broke authentication for my cross-vendor reviewer: the new CLI stopped reading the API key from the environment, and because the review step was fail-open, every adversarial review pass would have been skipped with no signal at all.

For anyone building agentic systems, these failures are not edge cases but inevitable operational realities. The economic cost of re-running merged work and the security risk of silently skipped reviews demand architectural answers, not prompt tweaks.

This article details the two mechanisms I implemented: a settled-state disposition check that classifies a task's PR before any re-queue (and refuses to relaunch on uncertainty), and self-healing Codex authentication at daemon startup that re-logins automatically and alarms me only when it cannot.

The Architecture of My Agent Fleet

To understand the failure modes, we must first define the system architecture. My agent fleet is a personal autonomous system designed to take tasks from a GitHub Projects board and carry them to a reviewed pull request without human intervention until the final merge.

Component Breakdown

The daemon polls the board for new tasks. The pipeline is segmented by role, and deliberately by vendor, to maximize adversarial review quality:

The Workflow

The data flow is structured:

GitHub Poller -> Code Write -> Claude Reviewer -> Codex CLI -> GitHub Merge

This architecture relies on cross-vendor authentication. The Builder and the Reviewer are Anthropic; the final Skeptic is OpenAI. The separation adds model diversity and makes correlated review failures less likely, but it introduces a critical dependency: the Codex CLI must hold a valid, authenticated session, or the last safety gate quietly disappears.

Failure Mode 1: Orphans That Were Not Orphans

The first incident involved the orphan-reclaim routine, which exists to recover genuinely lost work: a task claimed by an agent that died before pushing anything. On July 11, 2026, at about 03:35 UTC, it did the opposite. It re-queued two tasks that were already done.

The Mechanism of Failure

The night before, I had squash-merged two agent PRs by hand from the command line and deleted their branches. Both tasks stayed open on the board on purpose: the remaining checklist items were mine to do, not the agent's, so the PRs were deliberately not linked to auto-close their tasks. That is a documented convention in my fleet for partially completable work.

The reclaim logic then saw exactly what it was built to look for: an open task, board status In Progress, no running container, and no open agent PR. It concluded the work was lost, re-queued both tasks, and the daemon claimed them again. Two executor containers spun up and started redoing already-merged work, one of them an Opus run at high reasoning effort. I caught it from another session and stopped the containers by hand; the duplicate work never reached a second PR, but the tokens were spent.

Root Cause: An Ambiguous Signal

The root cause was not a stale flag or a missed timeout. Every input the reclaim logic read was accurate. The bug was in the interpretation: "no open agent PR" is an ambiguous signal that covers two opposite realities. It can mean the agent died before pushing anything (work lost, re-queue is correct), or it can mean the PR was merged externally and its branch deleted (work shipped, re-queue is the worst possible action).

In agentic orchestration this pattern generalizes. Any heuristic that infers "work pending" from the absence of an artifact will eventually fire on the case where the artifact is absent because the work is finished. The external system, not the daemon's internal model, holds the truth.

The Solution: Disposition Before Re-Queue

The fix classifies the settled state of the task's PR before any re-queue decision. A useful GitHub detail makes this cheap: a merged PR keeps its head branch name even after the branch is deleted, so querying PRs across all states by the agent's branch naming convention still finds it. This is the exact probe the daemon runs, one call per reclaim candidate:

gh pr list --repo / --head "agent/" \
  --state all --json state --jq '.[0].state // empty'

The result is MERGED, CLOSED, OPEN, or empty for no PR at all. I did not take the branch-name behavior on faith; another part of my controller already relied on the same query to detect externally merged PRs, so it was proven in production before the reclaim fix leaned on it.

The disposition logic now distinguishes five cases. Note that in the GitHub API "merged" is not a value of the open/closed state field; it is a separate signal you must query for explicitly:

Case 4 came out of the fleet's own cross-vendor review of the fix: the first version treated any API read error as "no PR" and would have re-queued possibly-shipped work on a transient failure, which is precisely the incident happening again by another path. The shipped fix carries hermetic tests for every one of the five branches, including an integration test asserting that a read error produces no reclaim, no board flip, and no notification.

Failure Mode 2: Silent Authentication in Cross-Vendor Flows

The second incident, two days earlier on July 9, involved the Codex CLI that powers the cross-vendor skeptic. My daemon's invocation path had relied on the CLI inheriting its API key from an environment variable. After upgrading from 0.142.5 to 0.144.1, that path stopped authenticating: every invocation began failing with a 401 "Missing bearer" error until I established a persisted login. To be precise, this is a claim about my setup, not about the CLI in general: current Codex documentation supports explicit API-key authentication, including per-invocation keys for headless exec calls; a long-running daemon simply fits the persisted-login flow better than passing a key on every call.

The Impact on System Integrity

In my architecture, the Codex pass is the final adversarial gate. The review step is fail-open by design: if the skeptic cannot run, the pipeline continues rather than deadlocking. That design choice, sensible in isolation, turned a broken login into an invisible one. Every skeptic pass would have been silently skipped, and I would have kept merging PRs that had never faced their strongest critic. I only caught it because I happened to run a manual smoke test after the upgrade.

I had already been bitten once by the same shape of failure, when a mistyped model name made the skeptic silently skip every review, so I recognized the pattern immediately: fail-open plus no health signal equals a gate that can vanish without anyone noticing.

Root Cause: A Dependency Changed Its Contract

The deeper lesson is that a vendor CLI's authentication and error-handling behavior is part of your API surface, even though no schema documents it. My daemon had verified the model's availability at startup (a scar from an earlier incident) but never the session's validity, because until this upgrade an environment variable was all the CLI needed. When the dependency moved the credential contract, the upstream system inherited a new silent failure mode it never opted into, and no test existed to catch it because the old contract had never failed.

The Solution: Self-Healing Login at Startup

I implemented self-healing authentication at daemon startup, not a hard stop. Before the polling loop begins, the daemon checks the CLI's login status. If the CLI reports it is not logged in and an API key is configured, the daemon pipes the key into the CLI's login command on stdin, logs the outcome, and moves on. Only if that automatic re-login fails does it alarm me over Telegram, and even then it fails open after alarming rather than blocking startup.

The check runs before the existing model availability probe, so a successful auto-login lets the probe pass in the same boot. The whole flow is tested against a stubbed CLI binary, so the login logic is exercised in CI without real credentials. One review finding sharpened it further: the CLI keeps its persisted state, including the auth file, in one default directory per OS user, so every daemon process running under the same account shares the same login and two instances could clobber each other's credentials. The fix gives each daemon instance its own isolated state directory.

A security boundary worth stating explicitly: the OpenAI key never enters the builder or executor containers. Those containers receive only task coordinates and a repo-scoped Git token; the key reaches exactly one place, the short-lived Codex CLI subprocess on the daemon host, which reviews the diff embedded in its prompt, read-only, with no repository checkout. Agents run repository-controlled code (builds, tests, hooks), and any secret visible in their environment must be assumed readable by that code.

Why self-healing instead of fail-fast? Because the failure is routine and the remediation is mechanical. Halting a fleet at 3 AM over an expired login trades availability for a ritual that a subprocess call performs better. The principle is: heal what you can heal automatically, alarm loudly on what you cannot, and never continue silently degraded.

Implementation Strategy: Two Reusable Patterns

The solutions above generalize beyond my stack. Here is how to operationalize them.

Pattern 1: Self-Healing Credential Verification

For any cross-vendor CLI or API dependency, add a startup gate that heals before it alarms.

Pseudocode for startup auth verification

if not cli_login_ok():
    if api_key_configured() and cli_login_with_key():
        log("auth self-healed at startup")
    else:
        alert("CRITICAL: cross-vendor reviewer not authenticated")
        # fail-open AFTER alarming: degraded, but never silently

Operational implication: the daemon can no longer boot into a state where the reviewer's credentials are already dead. Silence, not degradation, is the enemy.

Be honest about the limit of a startup gate: credentials can still expire, be revoked, or hit a vendor outage hours into a long-running process, and this check will not catch that. The next step on my roadmap is a structured per-invocation review status, where a skipped review is its own visible state on the PR instead of an absence of findings.

Pattern 2: Disposition Check in Reclaim Logic

Any task recovery mechanism must classify the external artifact before re-queuing.

Pseudocode for the reclaim decision

disposition = settled_pr_disposition(task)  # merged | rejected | closed | unknown | none
if disposition == "merged":
    flip_to_human(task); notify_once(task)
elif disposition in ("rejected", "closed"):
    surface(task)          # never silently relaunch
elif disposition == "unknown":
    skip_this_cycle(task)  # re-probe on the next scan
else:
    requeue(task)          # genuinely lost work

Operational implication: the observed false-reclaim path becomes fail-safe, including on transient API errors. It is not an absolute guarantee: the probe and the re-queue are two separate operations, so a PR merged in the window between them could still slip through, and a per-task lease or a second disposition check right before container launch would close that race. What the logic does guarantee is that no known settled state, and no unreadable state, ever relaunches work.

Challenges & Constraints

Actionable Takeaways

For CTOs, founders, and senior engineers building autonomous AI systems, here are the critical next steps:

FAQ

When did these incidents happen?

Both in July 2026: the Codex CLI authentication break on July 9 after upgrading from 0.142.5 to 0.144.1, and the orphan-reclaim incident on July 11 at about 03:35 UTC.

How much damage did the orphan-reclaim incident cause?

Two tasks were re-queued and two executor containers launched, one running Opus at high reasoning effort. I stopped the containers by hand before a duplicate PR was opened, so the cost was the wasted agent runtime and tokens, not repository damage.

What was the hardest part of the reclaim fix?

The uncertainty case. The obvious fix (re-queue unless the PR is merged) still re-queues shipped work whenever the status read fails transiently. The final logic treats an unreadable state as "unknown" and skips that cycle, then re-probes on the next scan instead of guessing.

Related reading

Sources