# A Publication Gate for AI-Written Articles: Score It, Revise It, Then Let a Human Press Publish

Author: Maksym Tytarenko | Date: 2026-07-13 | Category: Automation | Tags: ai content, quality gate, automated pipelines, content management, fact-checking
Canonical: https://www.tytarenkoagency.com/blog/publication-gate-ai-written-articles

> A three-layer quality gate for AI-generated articles: deterministic checks, a rubric-based LLM judge, brief-grounding verification, bounded revision, and final human approval. With real numbers from the gate's first run, which this very article failed at 5.0/10.

I added a three-layer quality gate to my automated blog pipeline so that “is this article good enough” became a number instead of a taste debate. To be precise, the gate is a hybrid, not a purely deterministic system: cheap deterministic checks, a rubric-based LLM score, brief-grounding fact findings, and a final human approval. Subjectivity is not eliminated; it is formalized into a rubric and moved to the end of the pipeline, where a human presses the button. Every draft now reaches my Telegram approval chat with an explicit score attached, and anything below the threshold carries a visible warning badge.

![An infographic that visually explains the three-layer quality gate system.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260713_040327_ac380ff7.png)


AI-written content pipelines scale faster than human review capacity, so low-quality drafts either flood editors or get published with hidden errors. For anyone running content automation, one fabricated number or truncated paragraph in published content costs far more than the inference behind a robust gate.

## The Problem: Subjective Quality in Automated Content Pipelines

When I first deployed an automated blog pipeline using Perplexity for research and GPT for cleanup, draft quality was wildly inconsistent. One draft would be a tight technical deep dive; the next, a ramble with leftover JSON fences, bracketed placeholders, and mid-sentence truncations. Without a gate, the only metric was “does this feel good,” a taste debate that wastes hours.

In agentic workflows, the bottleneck is rarely generation; it’s revision quality. If you don’t define pass/fail criteria and a bounded retry loop with a clear stop condition, your pipeline will spin forever or ship garbage.

In my pipeline, the gate runs before any image is created, so no money is wasted illustrating text that will be rewritten.

## Architecture: The Three-Layer Quality Gate

My gate is a three-layer system where a draft passes only when both the deterministic score and the LLM judge score clear the threshold, and there are zero critical grounding findings. A failing draft is never published automatically; it degrades to manual review (more on that below). The layers are:

1. **Layer 1: Deterministic Python Grader** (cost: $0)
2. **Layer 2: Single LLM Judge** (Claude Sonnet, different vendor from writer stack)
3. **Layer 3: Brief-Grounding Checker** (no web fetching, source of truth = topic description only)

### Layer 1: Deterministic Python Grader

This layer costs nothing and catches “generation junk” without spending a single token on evaluation. In my implementation the LLM layers still run in the same iteration (their findings feed the same revision call), so the saving is not in skipped LLM calls: a meaningful share of real defects gets caught by code that never guesses. The score starts at 10 and deducts points for:

![A detailed diagram showcasing the features of the deterministic Python grader.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260713_040250_55721550.png)


- **Em/en-dashes**: Hard gate. The grader rejects any draft with em or en dashes, using ASCII hyphens only.
- **Generation junk**: Leftover JSON fences, placeholder text, and mid-sentence truncations are hard gates.
- **Stop-phrase list**: A config-extensible list of phrases that immediately fail the draft.
- **Word count**: Target range for word count. Drafts outside this range fail.
- **Sentence-length variance**: Excluding code blocks, headings, and lists, the grader calculates standard deviation of sentence length. Monotone rhythm fails the draft.
- **FAQ contract**: 3 to 5 Q&A pairs with no links in answers. Missing or malformed FAQ fails.
- **H2 sections**: Minimum 4 H2 sections. Fewer fails.
- **Meta limits**: Title and description character limits.

This layer is deterministic: it doesn’t guess, it counts. A caveat: junk, truncation and malformed output measure quality; dash policy, FAQ shape, heading count and word range are house-style and template contracts. I keep them in one score because my pipeline targets one template, but a draft with three H2 sections is not automatically worse than one with four. A hard gate forces a revision regardless of the score.

### Layer 2: Single LLM Judge (Claude Sonnet)

Layer 2 is a single LLM call to Claude Sonnet, deliberately a different vendor from the writer stack (Perplexity + GPT). Cross-vendor judging reduces [self-preference bias](https://arxiv.org/abs/2410.21819), a documented failure mode of LLM judges; it does not make the judge objective, and rubric-based LLM evaluation like [G-Eval](https://arxiv.org/abs/2303.16634) still needs calibration against human judgments over time. The judge returns:

- A rubric score from 1 to 10, weighing technical depth and specificity (concrete examples, architectures, tradeoffs), clarity and flow, absence of filler and AI-tells, structural soundness, and whether the title’s promise is delivered
- Findings, each tagged with a category (depth, clarity, structure, tone, filler, accuracy)

The pass threshold of 8 is a borrowed starting value, not a derived one. It is deliberately env-tunable, and the first weeks of real drafts are the calibration data.

I deliberately did not copy the 5 parallel LLM critics from the public write-up that inspired this. Revision quality is the real bottleneck, and I didn’t yet know my pipeline’s actual failure modes. Instead, every finding is stored with its category per layer per iteration. Recurring categories will be promoted to dedicated critics later.


### Layer 3: Brief-Grounding Checker

Calling this layer a fact-checker would be generous, so let me name it precisely: it is a brief-grounding checker. It verifies the draft against the approved topic brief, not against the world. For first-person project stories, the only source of truth is the topic description: any number, date, or claim about the project not present there is a critical finding. There is no web fetching at all.

The honest limits follow directly from that design: a wrong number that is already in the brief sails through, and a misread external source will not be caught. Verifying external claims properly is a different discipline (decompose the text into atomic claims, tie each to evidence, check entailment; see DeepMind’s [SAFE](https://arxiv.org/abs/2403.18802) for what that takes). For first-person stories, brief-grounding is the right scope.

Fabricated numbers are the most expensive error class in published content: a single wrong number can invalidate an article’s credibility. This layer stops new numeric claims that were never in the approved brief from reaching the reader.

## Workflow: From Draft to Publish

The gate runs as follows:

1. **Draft generation**: Perplexity research + GPT cleanup → raw draft.
2. **Layer 1 check**: Python grader runs. A hard-gate hit forces a revision no matter how well the draft scores elsewhere.
3. **Layer 2 + Layer 3 parallel**: Claude Sonnet judge and brief-grounding checker run in parallel; their findings feed the same revision call, which is why they run even when Layer 1 already failed.
4. **Pass condition**: Deterministic score AND judge score clear the threshold, zero critical grounding findings, no hard gates.
5. **Revision loop**: If fail, a targeted GPT-4o revision fixes *only* the listed findings while preserving voice and length.
6. **Re-run**: After each revision, all three layers re-run.
7. **Degrade to manual review**: After 2 failed revisions, the draft ships to my Telegram approval chat with a warning score badge. The human keeps the final Publish button.

![A flowchart illustrating the steps in the AI content pipeline from generation to publication.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260713_040200_43f9f684.png)


A naming caveat: in the classic security sense, fail-open means a failed control lets the operation through. That is not what happens here. Nothing is published automatically; a failing draft degrades to manual review with a warning badge (“fail-to-human”). The only true fail-open is an evaluator outage (API timeout, malformed JSON): the draft proceeds to the same human gate unscored, with a logged warning, so an evaluator crash can never stall the pipeline.

### Pseudo-code for the Gate Loop

```python
def run_quality_gate(draft, topic_desc, threshold=8, max_revisions=2):
    for i in range(max_revisions + 1):
        det = deterministic_grader(draft)   # score + hard gates, $0

        try:
            judge, grounding = run_in_parallel(
                claude_judge(draft),
                brief_grounding_checker(draft, topic_desc),
            )
        except EvaluatorError as error:
            # Evaluator outage: ship unscored to the same human gate.
            log_evaluator_error(error)
            return {"status": "manual_review_required",
                    "reason": "evaluator_failure",
                    "draft": draft, "scores": None}

        passed = (det.score >= threshold
                  and judge.score >= threshold
                  and not grounding.has_criticals()
                  and not det.has_hard_gates())
        if passed or i == max_revisions:
            break

        # Targeted revision: fix ONLY the listed findings
        findings = det.findings + judge.findings + grounding.findings
        draft = revise(draft, findings)

    # Never auto-publish on failure: degrade to the human gate.
    return {"status": "passed" if passed else "manual_review_required",
            "draft": draft,
            "deterministic_score": det.score,
            "judge_score": judge.score,
            "critical_grounding_count": len(grounding.criticals)}
```

The `revise` function is a targeted GPT-4o call that fixes *only* the listed findings. It preserves voice and length, which is critical for maintaining article quality. Surface failures as per-layer numbers (deterministic, judge, critical grounding findings) rather than one blended score, so a grounding failure cannot hide behind two high scores; my own chat badge still shows a single number, and upgrading it is on the list.

## Implementation Example: My Blog Pipeline

### Context

The blog pipeline is part of this fleet: I use Perplexity for research, GPT for cleanup, and the three-layer gate before publishing. The goal is to automate content creation without sacrificing quality.

![A tech stack diagram showcasing the components of the AI blog pipeline.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260713_040119_cb03082e.png)


### Stack

- **Backend**: FastAPI service on Google Cloud Run
- **Storage**: Supabase (Postgres) for drafts and per-iteration eval records
- **Models**: Perplexity (research), GPT (cleanup), Claude Sonnet (judge and brief-grounding checker, separate calls), GPT-4o (revision)
- **Approval**: Telegram approval chat

### Workflow

1. **Topic seed**: a topic arrives from a Telegram command or an API call.
2. **Research**: Perplexity fetches context, returns raw notes.
3. **Cleanup**: GPT converts notes to a draft.
4. **Gate**: Three-layer quality gate runs, before any image generation.
5. **Revision**: If fail, GPT-4o revises only the findings.
6. **Re-run**: Gate re-runs after revision.
7. **Approval**: Draft ships to Telegram with a warning badge if the gate failed. Founder presses Publish.

### Outcome: The First Real Run

I only have one data point so far, but it is a telling one, because the first draft to go through the gate was an early version of this very article. It failed, and the failure report was legitimate:

- The grounding checker flagged 13 findings on the first pass, including 10 invented numeric claims, including an invented infrastructure story (a task queue and polling daemon this pipeline does not have): real fabrications, exactly the error class the layer was built for.
- The deterministic grader held the score at 5.0 across all three iterations. Two revisions cut em-dashes from 18 to 3 to 2 but could never reach zero, because an article about a junk filter has to quote junk patterns that the reviser cannot remove without destroying the paragraph. That calibration bug (exclude code spans from hard gates) is already filed.
- The judge scored 6.0, then 4.0, then 5.0 across iterations: revisions do not monotonically improve a draft, which is precisely why the loop is bounded.

The draft degraded to manual review at 5.0/10, I fixed the remaining issues by hand, and pressed Publish. The gate did its job: it turned "this feels off" into a specific, itemized defect list. And for completeness, the very first deploy attempt of the gate itself failed for a reason no eval can catch: GitHub Actions jobs dying in about 2 seconds with zero steps executed, because of a billing spending limit.

### Lessons Learned

- **Copy**: The deterministic layer. It catches defects with zero-cost code that never guesses, and reduces how much you must lean on subjective LLM judgment.
- **Avoid**: 5 parallel LLM critics. I didn’t know my failure modes yet, so I stored findings by category and will promote recurring categories to dedicated critics later.
- **Failure mode**: The brief-grounding checker got its own critic upfront because a fabricated number is the most expensive error class. This was the one exception to “data-driven critics.”

## Practical Implementation: How to Build Your Own Gate

### Step 1: Define Hard Gates

Start with a deterministic grader that catches:

- Leftover JSON fences
- Placeholder text
- Mid-sentence truncations
- Em/en-dashes (if you want ASCII-only)
- Stop phrases

These are hard gates: if any fire, the draft cannot pass, no matter how well it scores elsewhere, and goes into revision.

### Step 2: Choose Your LLM Judge

Pick a single LLM judge from a different vendor than your writer stack. For example:

- Writer stack: Perplexity + GPT
- Judge: Claude Sonnet

This reduces the risk of self-preference bias and adds model diversity, but it does not make the score objective. The judge returns a rubric score and categorized findings.

### Step 3: Ground the Draft in Its Brief

For first-person stories, the only source of truth is the topic description. No web fetching. Any number not in the topic description is a critical finding. Be honest about the scope: this checks provenance against the approved brief, not truth against the world, and that is the right trade for personal project stories.

### Step 4: Implement Bounded Retry

Max 2 revisions. After each revision, re-run all layers. If still fail, ship to human approval with a warning badge.

### Step 5: Store Findings by Category

Every finding is stored with its category per layer per iteration. Recurring categories will be promoted to dedicated critics later. This is data-driven: you don’t need 5 critics upfront; you need to know your failure modes first.

## Actionable Takeaways

- **Audit your current pipeline**: Identify where “generation junk” (placeholder text, JSON fences) appears. Add a deterministic grader to catch it first.
- **Prototype the deterministic layer first**: It’s cost-free and catches most junk. Don’t start with 5 LLM critics.
- **Surface critical fact findings upfront**: Fact-checking should be a separate critic because fabricated numbers are the most expensive error class.
- **Set an env-tunable threshold** and adjust it as you learn your failure modes.
- **Degrade to human approval, never auto-publish**: After 2 failed revisions, ship to Telegram with a warning badge. The human keeps the final Publish button.

## FAQ

### How do I build a deterministic grader for AI-written articles?

Start with a Python script that checks for hard gates: leftover JSON fences, placeholder text, mid-sentence truncations, and em/en-dashes. Add config-extensible stop-phrase lists, word count targets, sentence-length variance, FAQ contracts, and meta title/description limits. The score starts at 10 and deducts for each violation. This layer costs nothing and catches a large share of defects deterministically.

### Why use a different vendor for the LLM judge than the writer stack?

Using a different vendor (e.g., Claude Sonnet as judge vs. Perplexity + GPT as writer) reduces the risk of self-preference bias: a judge from the same family might score the draft higher because it is “familiar” with its own style. Cross-vendor judging adds model diversity, but the score still needs calibration against human judgments.

### What is the fail-open strategy in a quality gate?

In this pipeline, after 2 failed revisions the draft degrades to manual review: it ships to human approval (a Telegram chat) with a warning score badge, and the human keeps the final Publish button. Nothing is published automatically on failure, so "fail-to-human" is the more precise term than classic fail-open.

### How do I prevent fabricated numbers in AI-written content?

Isolate grounding as a separate critic. For first-person project stories, the only source of truth is the topic description. Any number not present there is a critical finding. There is no web fetching at all. To be precise about the guarantee: this blocks new numeric claims that were never in the approved brief, which are the most expensive error class in published content; it cannot catch a wrong number that already sits in the brief itself.

### When should I promote a recurring finding category to a dedicated critic?

Store every finding with its category per layer per iteration. After you’ve collected enough data, identify recurring categories. Promote those to dedicated critics. The exception is brief-grounding, which gets its own critic upfront because fabricated numbers are the most expensive error class.

## Sources

- [AI Agent Pipeline (Redis)](https://redis.io/blog/ai-agent-pipeline/)
- [I Built an SDLC Pipeline Where AI Agents Ship Features (LinkedIn)](https://www.linkedin.com/pulse/i-built-sdlc-pipeline-where-ai-agents-ship-features-heres-yassine-dwkte)
- [Quality Gates for Coding Agents: Stop Hooks Make Validation Mandatory](https://fbakkensen.github.io/ai/devtools/development/2026/03/27/quality-gates-for-coding-agents-how-stop-hooks-make-validation-mandatory.html)
- [Why Coding Agents Need Independent Quality Gates (Codacy)](https://blog.codacy.com/why-coding-agents-need-independent-quality-gates)
- [Vibe Coding Quality Gate: CI/CD for AI-Generated PRs (Autonoma)](https://getautonoma.com/blog/quality-gate-vibe-coding)
- [Building a Five-Layer Quality Gate for Agent-Written Code (dev.to)](https://dev.to/kagin007/building-a-five-layer-quality-gate-for-agent-written-code-3e0k)
- [Pipeline Quality Gates (InfoQ)](https://www.infoq.com/articles/pipeline-quality-gates/)
- [Quality Gate (SonarSource)](https://www.sonarsource.com/resources/library/quality-gate/)

---

Transparency note: this article was produced by the exact pipeline it describes. Research was assisted by Perplexity, the draft was written and revised through the three-layer gate (where it initially failed at 5.0/10), and it was then hand-edited, fact-checked, and approved by Maksym Tytarenko.