Enhancing AI Review Processes: From Effort Allocation to Test Evidence

I built agent-corp as my founder-side autonomous AI agent fleet: a daemon on a GCP VM polls a GitHub Projects board and runs each task as headless Claude Code inside a Docker container. The newest change is not about making the fleet “smarter” in a vague sense; it is about making review more disciplined by assigning a reasoning-effort level to each container role and requiring reviewers to paste real test-runner evidence before a pull request can move forward.

That shift matters because AI review systems fail in predictable ways. A reviewer model can sound confident while missing a defect, and it can also claim that tests ran when no verifiable evidence exists. In an agentic code workflow where Claude builds, Claude reviews, an optional GPT skeptic tries to refute, and I merge via Telegram buttons, the weak point is not just model quality; it is the audit trail. The point of this change is to make review decisions more inspectable, more resistant to false claims, and easier for a solo founder to trust at scale.

This article is for CTOs, founders, and technical leaders who are using or evaluating AI-assisted delivery pipelines. If your team is already comfortable with CI/CD, containerized execution, and LLM-based coding, the interesting question is no longer whether an agent can write code. It is whether your review process can prove what happened, under what scrutiny, and with what evidence.

Why review needed a stricter operating model

The original temptation with AI-assisted review is to treat all model calls as equivalent: one prompt in, one answer out, then humans decide whether to trust it. That works for prototypes, but it becomes brittle once the system is responsible for code changes that may touch security-sensitive paths, deployment logic, or integration contracts.

In my setup, the task pipeline already had a clear division of labor:

The problem was not the existence of these stages; it was the uniformity of scrutiny. A routine formatting change and a security-related change should not receive the same review effort. Likewise, “tests passed” is not meaningful unless the reviewer can point to evidence that is independent of its own narration. A review process that depends on model self-reporting is too easy to bluff.

The architectural response is straightforward: make scrutiny explicit, role-specific, and evidence-backed. Each container role now receives a reasoning-effort level, and the review gate refuses to advance unless the reviewer includes pasteable, verifiable output from the real test runner.

The evening a placebo skeptic burned $93

None of this stayed theoretical. The forcing incident: one evening the cross-vendor skeptic started refuting every single first-pass pull request in the fleet. Full diffs were “reviewed” in two to five seconds, always with generic blocking findings. Every refutation escalated an extra revise cycle on the more expensive review models. By the time I looked at the logs, the fleet had burned roughly $93 of inference in one evening with zero merges.

The root cause was configuration, not model quality. The Codex CLI defaults to a reasoning effort of none in headless runs, and the session logs showed the skeptic spending exactly zero reasoning tokens per verdict. A model that is handed a full diff, prompted to assume the pull request is subtly wrong, and given no deliberation budget will always find something to object to. Those verdicts were a distribution artifact, not reviews.

It got worse before it got better: the CLI version we ran silently dropped the config override that was supposed to raise the effort. The startup banner printed “reasoning effort: high” while the actual API turn still ran at the default. The only place the truth appeared was the per-turn token usage in the session rollout files. After upgrading the CLI and pinning the effort in versioned configuration, the same command produced hundreds of reasoning tokens per verdict — and the skeptic went back to approving sound pull requests instead of refuting everything on sight.

Two lessons came out of that evening. First, reasoning effort is a real, per-call API parameter, and in a fleet it has to be pinned per role in orchestration config and then verified at the turn level, from token usage — not trusted from a banner or a prompt. Second, when a judge behaves suspiciously uniformly — always refutes, always within seconds — audit its effective configuration before you touch its prompt.

Per-role reasoning effort as an architectural control

By “reasoning effort” I mean the provider-side deliberation budget — the reasoning-effort parameter on OpenAI models, the extended-thinking budget on Claude — not a prompt convention that asks the model to “think carefully.” It is set where the orchestrator launches each container role, and the effective value is logged per turn, because the incident above proved that the requested and effective values can diverge.

The key idea is to stop treating the executor, reviewer, and adversarial skeptic as one generic “agent” and instead tune them as different operational roles:

A low-risk role can run with lighter scrutiny and a narrower context, while a review or security pass gets the larger budget because it is the last model-layer checkpoint before a human merge decision.

What this changes in practice

A concrete example is a pull request that touches authentication middleware. The executor can make the change, but the reviewer should operate under a higher-effort profile because the risk is not only functional failure but also policy drift, silent bypasses, or permission regressions. A low-effort review might confirm syntax and surface-level logic; a high-effort review inspects the diff adversarially, asks what assumptions changed, and forces a revise cycle if the implementation is under-specified.

A cosmetic refactor still deserves review, but not the same deliberative intensity as a code path that modifies request authorization or shell execution. Per-role effort lets the system spend attention where it matters instead of overpaying for the easy cases. The tradeoff is cost and latency: higher scrutiny increases inference time and spend, but it reduces the odds that a superficially plausible answer slips through the gate.

Why test evidence has to be real, not narrated

The second change is more important than it may sound: reviewers now have to paste real test-runner evidence. That requirement addresses a failure mode that is easy to miss in AI workflows: the model can claim the tests executed successfully even when it did not actually run them, or when it is only paraphrasing a partial output from memory.

If your review process accepts a textual claim like “tests passed” without attached output, the system is trusting narration instead of evidence. The fix is that the review step is simply not complete unless it contains the actual output produced by the execution environment.

A realistic workflow

What the gate can and cannot prove

Honesty matters here: this gate is a discipline mechanism, not a cryptographic proof of execution. What it validates is shape — the review must contain verbatim runner output with the command line, per-test results, and summary counts, rather than a prose summary. What it cannot validate is provenance: a model determined to fabricate plausible-looking runner output could still pass a textual check.

Three things make fabrication a bad strategy in practice. The tests run in the same container the reviewer already inhabits, so producing real output is cheaper than forging it. The pasted output becomes part of the audit trail, where it can be re-run and diffed if a merge is ever disputed. And the adversarial pass plus the human merge decision still sit behind the gate. The design goal is not absolute proof; it is raising the cost of bluffing from one confident sentence to a consistent forged artifact that has to survive later scrutiny.

That is also why the evidence matters beyond postmortems. When I scan a pull request, I need to know whether the review reflects actual execution or model-generated confidence. If a reviewer is wrong, the incorrectness is visible in the evidence chain — and that visibility changes behavior: the system rewards verification over rhetorical certainty.

The review loop, end to end

The full flow is:

Task board item -> GCP VM daemon -> Dockerized task container -> Claude executor (standard effort) -> Claude reviewer (high effort) -> revise cycle if needed -> optional cross-vendor GPT skeptic via Codex CLI (high effort, refute-only) -> founder merge via Telegram buttons -> audit log

The flow separates execution from approval. The executor is not the reviewer, the reviewer is not the final approver, and the final approver is not anonymous automation. The most important boundary is between “model output” and “system action”: models can propose, critique, and refute, but nothing crosses into a production change without the explicit approval gate. The merge button in Telegram is not a convenience detail; it is the final policy checkpoint. The skeptic, in turn, is refute-only by design — it never writes code — which keeps it a contradiction check rather than another authoring source.

Practical implementation layer

A production-grade implementation should treat this as a policy-driven workflow rather than a prompt convention.

Component responsibilities

Critical-path pseudo-code

def process_task(task):
    container = launch_container(task)
    code_change = claude_execute(
        container=container,
        effort="standard"
    )
    review = claude_review(
        container=container,
        diff=code_change.diff,
        effort="high"
    )
    if not review.contains_real_test_evidence():
        return request_revise(task, reason="missing verifiable test output")
    if review.needs_revision:
        code_change = claude_revise(
            container=container,
            feedback=review.feedback,
            effort="standard"
        )
        review = claude_review(
            container=container,
            diff=code_change.diff,
            effort="high"
        )
    if task.requires_cross_vendor_skeptic:
        refutation = codex_refute(
            container=container,
            diff=code_change.diff,
            effort="high"
        )
        if refutation.finds_material_issue:
            return request_revise(task, reason=refutation.findings)
    send_to_telegram_approval(task, code_change, review)

Two details make this instructive rather than illustrative. The effort values are not labels: at dispatch they map to the provider's actual reasoning parameter for that call. And the audit log records the effective per-turn effort read back from the session logs, not the requested one — because, as the skeptic incident showed, the two can silently diverge. Evidence checking is a gate, not a suggestion; review effort is a role property, not a free-form prompt.

Challenges and constraints

The main technical bottleneck is not raw model capability; it is orchestration reliability. Container startup time, context management, revision loops, and test execution all add latency, and higher-effort review adds more. That is acceptable for review and security passes, but the orchestration layer has to distinguish deliberate approval stages from time-sensitive execution — otherwise scrutiny becomes a bottleneck instead of a safeguard. If the review path becomes too slow, the founder stops trusting it as a daily workflow.

Cost is the second constraint: higher-effort review and adversarial refutation consume more inference than a single-pass assistant, and logging role decisions, evidence artifacts, and review traces adds overhead of its own. The $93 evening cuts both ways here — misconfigured scrutiny is expensive too, and that spend went to escalated revise cycles a correctly configured skeptic would never have triggered.

The security stance is simple: any autonomous review workflow should assume prompt injection, hallucination, and false confidence as baseline risks, then design controls around them. Evidence requirements and role-specific scrutiny are part of that control plane, not optional polish.

What CTOs should take away from this design

FAQ

Why does an AI code review need real test evidence?

Because a model can claim that tests ran without actually proving it. Requiring pasted test-runner output forces the review process to anchor its approval in observable execution rather than narration. The gate cannot prove provenance, but it turns approval into an auditable artifact and makes bluffing more expensive than just running the tests.

What does “per-role reasoning effort” mean in practice?

The executor, reviewer, and adversarial skeptic do not run with the same deliberation budget. Effort is a real per-call API parameter, pinned per container role in orchestration config, and verified per turn from token usage — because a requested “high” can silently run as “none.”

Why use a cross-vendor skeptic at all?

A different model family can surface different blind spots, especially when it is used only to refute the pull request. In this system the GPT/Codex step is adversarial only; it never writes code. That makes it a final contradiction check rather than another authoring source — provided its reasoning budget is actually configured, which is exactly what failed in the incident above.

Related reading