# Optimizing Agent Role Prompts Offline: What SkillOpt Automates, and the Manual Loop My Fleet Already Runs

Author: Maksym Tytarenko | Date: 2026-07-21 | Category: AI & ML | Tags: ai agents, prompt optimization, skillopt, autonomous agents, llmops
Canonical: https://www.tytarenkoagency.com/blog/optimizing-agent-role-prompts-offline-what-skillopt-automates-and-the-manual-loop-my-fleet-already-runs

> Microsoft's SkillOpt treats an agent's skill file as a trainable parameter, optimized offline from scored rollouts. Mapping it onto my own agent fleet revealed I already run a manual version of the loop: a per-repo lessons file distilled from reviewer findings and a weekly synthesis job that proposes skill edits. This article shows the real artifacts, what maps to the paper, and where automated validation beats a human gate.

# Optimizing Agent Role Prompts Offline: What SkillOpt Automates, and the Manual Loop My Fleet Already Runs

In May 2026 Microsoft Research published SkillOpt (arXiv 2605.23904), a framework that treats an agent's skill file as a trainable parameter. Instead of fine-tuning model weights, a separate optimizer model reads scored execution rollouts and proposes bounded add, delete, or replace edits to a single skill document. An edit is accepted only if it strictly improves performance on a held-out validation set. Across benchmarks spanning question answering, multi-round code generation with tool use, and document reasoning, the paper reports absolute accuracy gains of 15 to 25 points over a fixed hand-written skill, with all optimization cost paid offline, outside the live agent loop.

I run a small fleet of autonomous coding agents for my own projects, so my first question was practical: do I need to build this? While mapping SkillOpt onto my setup, I realized something more useful for a writeup. I already run a manual version of this exact loop, and the places where my version diverges from the paper are the places where the paper earns its complexity. This article walks through the real loop, with the real artifacts, and is honest about what I have measured and what I have not.

## The Fleet, Briefly

My production setup is a stateless daemon on a small Google Cloud VM. It polls a GitHub Projects board for tasks, and runs each claimed task as a headless Claude Code session inside a per-task Docker container. A cheap classifier model routes each task to an executor tier, so simple tasks do not burn frontier-model tokens. Every pull request then passes through review: a fresh reviewer session (deliberately one model tier below the executor, and with no memory of writing the code) checks the diff, with up to two revise cycles that escalate the executor model on each return. On top of that, a cross-vendor skeptic (the OpenAI Codex CLI) tries to refute the PR, on the theory that a different vendor's model does not share the author model's blind spots. The final merge is never automatic: it happens through approval buttons in Telegram, pressed by me.

![A visual representation of the fleet setup and its components.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260721_190403_ee441720.png)


The part that matters for this article: every role in that pipeline (executor, reviewer, skeptic) is defined by a prompt or skill file that lives in a git repository, versioned and diffed like code. That is the precondition for any SkillOpt-style loop. You cannot optimize a parameter you cannot version.

## The Loop That Actually Exists: A Lessons File

The static-prompt problem is real. My reviewer kept flagging the same classes of mistakes in fresh agent sessions, because each session starts blank and the role prompt cannot enumerate every pitfall of every repo.

The fix I converged on is embarrassingly simple. Each repo the fleet works in carries a lessons file: a flat list of one-line guidelines distilled from past reviewer findings, newest last. Executor agents read the tail of that file before starting any task. The file in my main production repo currently holds 43 lines. Real examples, verbatim in spirit:

![An infographic that summarizes the lessons file and its usage.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260721_190204_2ce4c7be.png)


- Supabase 2.0.0 query builders do not support the .or_() method; use the filter form instead.
- When an external operation succeeds but the local record write fails, mark the row failed or backed off, never scheduled, or the retry sweep will publish a duplicate.
- Wrap an entire async sequence that shares one deadline in a single asyncio.wait_for, not per-stage waits.
- Do not write test assertions that accept multiple outcomes; an assertion must reject at least one wrong result.

Here is one worked example of a trace becoming a guideline. A scheduled social post published successfully, but the write recording that publish failed. The status stayed scheduled, so the periodic sweep picked the row up again and published the same content twice. The reviewer findings from the fix distilled into three lessons file lines: the mark-failed-not-scheduled rule above, a rule to test the specific failure path where publish succeeds and the log write fails, and a rule that sweep loops need explicit retry-count backoff rather than status-only logic. Tasks touching the publishing rail read those lines before writing code, and that entire bug class has not come back through review since.

Map that onto the paper's vocabulary and the structure is the same shape. Scored rollouts: reviewer verdicts, skeptic refutations, and merge outcomes. Bounded edits: one appended line per finding, which is a far tighter bound than freeform prompt rewriting. Slow update: lines are added when review produces a finding, not continuously. Optimizer-side memory: the file itself is the memory, and newest-last ordering plus tail reading keeps the freshest guidance inside the context budget.

## The Second Loop: Weekly Synthesis

The lessons file captures repo-level pitfalls. A second, slower loop targets the role prompts and skills themselves.

![A visual depiction of the weekly synthesis process and its components.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260721_190100_27e4b876.png)


Daily, a cheap-model job digests my working sessions and the agent channels into a short signal-versus-noise summary. Weekly, a synthesis job reads the accumulated digests together with an inventory of my current skill files and files concrete improvement proposals as issues against the repo where the skills live: this instruction is being ignored, this skill is missing a step agents keep stumbling on, these two skills contradict each other. After synthesis, the raw digest corpus is deleted for privacy; only the distilled proposals persist. I review each proposal and commit the accepted skill edits by hand.

That weekly job is the closest thing I have to SkillOpt's optimizer model: a separate, cheaper model turning execution evidence into proposed edits on versioned prompt documents. The difference is acceptance. SkillOpt accepts an edit when it beats a held-out validation set. I accept an edit when it survives my own review. That substitution is the honest core of this article, so it deserves its own section.

## What I Have Not Measured, and Why the Paper's Validation Step Matters

I have no benchmark for my loop. The 15 to 25 point gains are SkillOpt's numbers on SkillOpt's benchmarks, not mine. My evidence is directional: recurring finding classes stop recurring after their lessons land, and the reviewer stops repeating itself. I will not dress that up as a percentage.

Two incidents taught me why human-only acceptance does not scale, and both come with real numbers.

First, the runaway. One of my orchestrated multi-agent runs, on what was a minor audit task, spawned roughly 287 subagents and burned about 15.3 million tokens before I noticed. The fix was not a better prompt. It was a static guard hook that parses any planned workflow before launch and blocks it when the declared fan-out exceeds 20 agents unless I explicitly acknowledge the scale. The lesson generalizes to prompt optimization: a self-improvement loop needs hard limits enforced outside the model, because the model that wrote the bad plan will not veto it. SkillOpt's bounded-edit constraint is the same instinct applied to text edits.

Second, the theater gate. I once added a mandatory comprehension quiz I had to pass before merging any agent-written PR. On a real change I failed it zero out of four, and merged anyway, because the work was fine and the quiz was measuring recall of details that did not matter. I deleted the gate that week. A validation step that does not change decisions is not validation, it is ritual. This is exactly the argument for SkillOpt's design: acceptance tied to a measured score on held-out tasks is cheap to run and actually binding, unlike a human ceremony that gets overridden the first time it is inconvenient.

So my current position: human review of prompt edits is the right gate at my scale (a handful of edits per week), and it stops being the right gate the moment edit volume grows past what I will genuinely evaluate. The upgrade path is not more discipline, it is an automated acceptance test.

## Cost Shape

Everything above runs offline, and on the cheapest model that does the job. The routing ladder in my fleet is explicit: a small fast model does reading and classification, a mid-tier model does bulk uniform work like digesting and summarizing, and strong models are reserved for verification and final judgment. The daily digest and weekly synthesis jobs run on the cheap tiers. The live task loop pays nothing for any of this; executors just read a few hundred extra tokens of lessons tail. That cost asymmetry, spend on optimization offline so live execution gets cheaper and fails less, is the property SkillOpt formalizes.

One honest caveat on file growth: an append-only lessons file gets stale. Mine is young enough that its lines fit comfortably, but the maintenance story at 400 lines is consolidation and deletion, which is worth noticing in the paper too: SkillOpt's edit space includes delete and replace, not just add. An optimizer that can only append will bloat its own context budget until guidance drowns in it.

## Takeaways

- Version your role prompts and skills in git before anything else. An unversioned prompt cannot be optimized, rolled back, or reasoned about.
- Start with a lessons file per repo: one line per reviewer finding, newest last, agents read the tail. It is a working SkillOpt skeleton you can build in an afternoon.
- Keep edits bounded. Appending a one-line guideline is reviewable in seconds; a freeform prompt rewrite is not.
- Put evidence before edits. A guideline earns its line when a finding actually recurred, not when someone imagines it might.
- Keep a human on the merge while edit volume is low, and be honest about when volume outgrows your attention. That is the point to automate acceptance against a held-out task set, which is the step SkillOpt contributes.
- Enforce hard limits outside the model: fan-out caps, spend caps, blocking hooks. Self-improving systems inherit their own blind spots.

![A visually appealing summary of the key takeaways from the article.](https://adltqzpaelnsrebtkzfj.supabase.co/storage/v1/object/public/blog-images/ai-generated/uploads/20260721_190006_e0514804.png)


## FAQ

### Does this require running an agent fleet?

No. A solo developer using one coding agent gets the same effect from a versioned per-repo instructions file that accumulates one-line lessons from code review. The fleet only adds volume, which makes the recurring patterns visible faster.

### Why edit prompts instead of fine-tuning the model?

Prompt and skill edits are inspectable, diffable, instantly deployable, and instantly reversible, and they survive a model upgrade. Fine-tuning is none of those things.

### When should acceptance be automated?

When the number of proposed edits per week exceeds what you will honestly evaluate one by one. At that point, build a small held-out set of representative tasks and accept an edit only when it improves the pass rate.

### How do you keep the guidance from outgrowing the context window?

Bound what agents read (the tail, not the file), and treat deletion as a first-class edit. Consolidate overlapping lines and remove guidance that stopped being relevant, otherwise the file slowly crowds out the task itself.

## Related reading

- [Don't Install GitHub Spec Kit — Steal These Three Ideas Instead](https://www.tytarenkoagency.com/blog/dont-install-spec-kit-steal-ideas)
- [From Hype to Reality: Managing Agentic AI Expectations and Delivering Actual Value in 2026](https://www.tytarenkoagency.com/blog/from-hype-to-reality-managing-agentic-ai-expectations-and-delivering-actual-value-in-2026)
- [Context Engineering for Vibe Coding: Structured Prompts for Rapid SaaS Prototyping](https://www.tytarenkoagency.com/blog/context-engineering-for-vibe-coding-structured-prompts-for-rapid-saas-prototyping)

## Sources

- [microsoft.com](https://www.microsoft.com/en-us/research/blog/skillopt-agent-skills-as-trainable-parameters/)
- [huggingface.co](https://huggingface.co/papers/2605.23904)
- [venturebeat.com](https://venturebeat.com/orchestration/microsofts-open-source-skillopt-automatically-upgrades-ai-agent-skills-without-touching-model-weights)