A Publication Gate for AI-Written Articles: Score It, Revise It, Then Let a Human Press Publish

I added a three-layer quality gate to my automated blog pipeline so that “is this article good enough” became a number instead of a taste debate. To be precise, the gate is a hybrid, not a purely deterministic system: cheap deterministic checks, a rubric-based LLM score, brief-grounding fact findings, and a final human approval. Subjectivity is not eliminated; it is formalized into a rubric and moved to the end of the pipeline, where a human presses the button. Every draft now reaches my Telegram approval chat with an explicit score attached, and anything below the threshold carries a visible warning badge.

AI-written content pipelines scale faster than human review capacity, so low-quality drafts either flood editors or get published with hidden errors. For anyone running content automation, one fabricated number or truncated paragraph in published content costs far more than the inference behind a robust gate.

The Problem: Subjective Quality in Automated Content Pipelines

When I first deployed an automated blog pipeline using Perplexity for research and GPT for cleanup, draft quality was wildly inconsistent. One draft would be a tight technical deep dive; the next, a ramble with leftover JSON fences, bracketed placeholders, and mid-sentence truncations. Without a gate, the only metric was “does this feel good,” a taste debate that wastes hours.

In agentic workflows, the bottleneck is rarely generation; it’s revision quality. If you don’t define pass/fail criteria and a bounded retry loop with a clear stop condition, your pipeline will spin forever or ship garbage.

In my pipeline, the gate runs before any image is created, so no money is wasted illustrating text that will be rewritten.

Architecture: The Three-Layer Quality Gate

My gate is a three-layer system where a draft passes only when both the deterministic score and the LLM judge score clear the threshold, and there are zero critical grounding findings. A failing draft is never published automatically; it degrades to manual review (more on that below). The layers are:

Layer 1: Deterministic Python Grader

This layer costs nothing and catches “generation junk” without spending a single token on evaluation. In my implementation the LLM layers still run in the same iteration (their findings feed the same revision call), so the saving is not in skipped LLM calls: a meaningful share of real defects gets caught by code that never guesses. The score starts at 10 and deducts points for:

This layer is deterministic: it doesn’t guess, it counts. A caveat: junk, truncation and malformed output measure quality; dash policy, FAQ shape, heading count and word range are house-style and template contracts. I keep them in one score because my pipeline targets one template, but a draft with three H2 sections is not automatically worse than one with four. A hard gate forces a revision regardless of the score.

Layer 2: Single LLM Judge (Claude Sonnet)

Layer 2 is a single LLM call to Claude Sonnet, deliberately a different vendor from the writer stack (Perplexity + GPT). Cross-vendor judging reduces self-preference bias, a documented failure mode of LLM judges; it does not make the judge objective, and rubric-based LLM evaluation like G-Eval still needs calibration against human judgments over time. The judge returns:

The pass threshold of 8 is a borrowed starting value, not a derived one. It is deliberately env-tunable, and the first weeks of real drafts are the calibration data.

I deliberately did not copy the 5 parallel LLM critics from the public write-up that inspired this. Revision quality is the real bottleneck, and I didn’t yet know my pipeline’s actual failure modes. Instead, every finding is stored with its category per layer per iteration. Recurring categories will be promoted to dedicated critics later.

Layer 3: Brief-Grounding Checker

Calling this layer a fact-checker would be generous, so let me name it precisely: it is a brief-grounding checker. It verifies the draft against the approved topic brief, not against the world. For first-person project stories, the only source of truth is the topic description: any number, date, or claim about the project not present there is a critical finding. There is no web fetching at all.

The honest limits follow directly from that design: a wrong number that is already in the brief sails through, and a misread external source will not be caught. Verifying external claims properly is a different discipline (decompose the text into atomic claims, tie each to evidence, check entailment; see DeepMind’s SAFE for what that takes). For first-person stories, brief-grounding is the right scope.

Fabricated numbers are the most expensive error class in published content: a single wrong number can invalidate an article’s credibility. This layer stops new numeric claims that were never in the approved brief from reaching the reader.

Workflow: From Draft to Publish

The gate runs as follows:

A naming caveat: in the classic security sense, fail-open means a failed control lets the operation through. That is not what happens here. Nothing is published automatically; a failing draft degrades to manual review with a warning badge (“fail-to-human”). The only true fail-open is an evaluator outage (API timeout, malformed JSON): the draft proceeds to the same human gate unscored, with a logged warning, so an evaluator crash can never stall the pipeline.

Pseudo-code for the Gate Loop

def run_quality_gate(draft, topic_desc, threshold=8, max_revisions=2):
    for i in range(max_revisions + 1):
        det = deterministic_grader(draft)   # score + hard gates, $0
        try:
            judge, grounding = run_in_parallel(
                claude_judge(draft),
                brief_grounding_checker(draft, topic_desc),
            )
        except EvaluatorError as error:
            # Evaluator outage: ship unscored to the same human gate.
            log_evaluator_error(error)
            return {"status": "manual_review_required",
                    "reason": "evaluator_failure",
                    "draft": draft, "scores": None}
        passed = (det.score >= threshold
                  and judge.score >= threshold
                  and not grounding.has_criticals()
                  and not det.has_hard_gates())
        if passed or i == max_revisions:
            break
        # Targeted revision: fix ONLY the listed findings
        findings = det.findings + judge.findings + grounding.findings
        draft = revise(draft, findings)
    # Never auto-publish on failure: degrade to the human gate.
    return {"status": "passed" if passed else "manual_review_required",
            "draft": draft,
            "deterministic_score": det.score,
            "judge_score": judge.score,
            "critical_grounding_count": len(grounding.criticals)}

The revise function is a targeted GPT-4o call that fixes only the listed findings. It preserves voice and length, which is critical for maintaining article quality. Surface failures as per-layer numbers (deterministic, judge, critical grounding findings) rather than one blended score, so a grounding failure cannot hide behind two high scores; my own chat badge still shows a single number, and upgrading it is on the list.

Implementation Example: My Blog Pipeline

Context

The blog pipeline is part of this fleet: I use Perplexity for research, GPT for cleanup, and the three-layer gate before publishing. The goal is to automate content creation without sacrificing quality.

Stack

Workflow

Outcome: The First Real Run

I only have one data point so far, but it is a telling one, because the first draft to go through the gate was an early version of this very article. It failed, and the failure report was legitimate:

The draft degraded to manual review at 5.0/10, I fixed the remaining issues by hand, and pressed Publish. The gate did its job: it turned "this feels off" into a specific, itemized defect list. And for completeness, the very first deploy attempt of the gate itself failed for a reason no eval can catch: GitHub Actions jobs dying in about 2 seconds with zero steps executed, because of a billing spending limit.

Lessons Learned

Practical Implementation: How to Build Your Own Gate

Step 1: Define Hard Gates

Start with a deterministic grader that catches:

These are hard gates: if any fire, the draft cannot pass, no matter how well it scores elsewhere, and goes into revision.

Step 2: Choose Your LLM Judge

Pick a single LLM judge from a different vendor than your writer stack. For example:

This reduces the risk of self-preference bias and adds model diversity, but it does not make the score objective. The judge returns a rubric score and categorized findings.

Step 3: Ground the Draft in Its Brief

For first-person stories, the only source of truth is the topic description. No web fetching. Any number not in the topic description is a critical finding. Be honest about the scope: this checks provenance against the approved brief, not truth against the world, and that is the right trade for personal project stories.

Step 4: Implement Bounded Retry

Max 2 revisions. After each revision, re-run all layers. If still fail, ship to human approval with a warning badge.

Step 5: Store Findings by Category

Every finding is stored with its category per layer per iteration. Recurring categories will be promoted to dedicated critics later. This is data-driven: you don’t need 5 critics upfront; you need to know your failure modes first.

Actionable Takeaways

FAQ

How do I build a deterministic grader for AI-written articles?

Start with a Python script that checks for hard gates: leftover JSON fences, placeholder text, mid-sentence truncations, and em/en-dashes. Add config-extensible stop-phrase lists, word count targets, sentence-length variance, FAQ contracts, and meta title/description limits. The score starts at 10 and deducts for each violation. This layer costs nothing and catches a large share of defects deterministically.

Why use a different vendor for the LLM judge than the writer stack?

Using a different vendor (e.g., Claude Sonnet as judge vs. Perplexity + GPT as writer) reduces the risk of self-preference bias: a judge from the same family might score the draft higher because it is “familiar” with its own style. Cross-vendor judging adds model diversity, but the score still needs calibration against human judgments.

What is the fail-open strategy in a quality gate?

In this pipeline, after 2 failed revisions the draft degrades to manual review: it ships to human approval (a Telegram chat) with a warning score badge, and the human keeps the final Publish button. Nothing is published automatically on failure, so "fail-to-human" is the more precise term than classic fail-open.

How do I prevent fabricated numbers in AI-written content?

Isolate grounding as a separate critic. For first-person project stories, the only source of truth is the topic description. Any number not present there is a critical finding. There is no web fetching at all. To be precise about the guarantee: this blocks new numeric claims that were never in the approved brief, which are the most expensive error class in published content; it cannot catch a wrong number that already sits in the brief itself.

When should I promote a recurring finding category to a dedicated critic?

Store every finding with its category per layer per iteration. After you’ve collected enough data, identify recurring categories. Promote those to dedicated critics. The exception is brief-grounding, which gets its own critic upfront because fabricated numbers are the most expensive error class.

Sources

Transparency note: this article was produced by the exact pipeline it describes. Research was assisted by Perplexity, the draft was written and revised through the three-layer gate (where it initially failed at 5.0/10), and it was then hand-edited, fact-checked, and approved by Maksym Tytarenko.