# Hardening Docker CLI Shell-Outs for a More Resilient Autonomous AI Agent Fleet

Author: Maksym Tytarenko | Date: 2026-07-28 | Category: AI & ML | Tags: docker, cli, autonomous agents, exception safety, time bounding, ai systems, resilience
Canonical: https://www.tytarenkoagency.com/blog/hardening-docker-cli-autonomous-ai-fleet

> One hung Docker socket taught me to treat every CLI shell-out in my agent fleet as a production dependency: 10-second timeouts, fail-open telemetry, fail-closed isolation, and an adversarial review that found the gaps I missed.

# Hardening Docker CLI for Resilient AI Agent Fleets

I run an autonomous agent fleet called **agent-corp**: a daemon on a GCP VM polls a GitHub Projects board and launches every accepted task as headless Claude Code inside a Docker container. This week I shipped a change that made every Docker CLI shell-out on that launch path **exception-safe** and **time-bounded**. A missing Docker binary or a hung Docker socket now degrades gracefully instead of aborting the launch or freezing a thread forever.

The interesting part is not the try/except blocks. It is how the gaps were found (an adversarial review pass on a fix I already believed was complete), and the semantics I had to pin down along the way: when a Docker failure should **fail open** and when it must **fail closed**. This is a story about treating a shell-out as a production dependency, with real code and real numbers.

## The incident that started it

The chain began with a telemetry outage, not a crash. A deploy restarted the fleet daemon and its egress proxy within the same few seconds. The daemon's startup probe checked the OTLP telemetry path, hit the proxy before it had finished coming up, and cached the verdict: telemetry path broken. From that moment every task launched without telemetry env injected. The path itself was fine minutes later, but nothing ever re-checked the cached verdict.

The failure was silent. No task failed; work kept shipping. I only found out the next day when an external uptime alert fired on missing executor telemetry. Recovery was a manual daemon restart, after which the very same probe passed.

The immediate fix was obvious: re-probe with backoff during startup, re-check before launches instead of trusting a startup-cached verdict, log the flip from broken to healthy and resume telemetry without a restart, and send a Telegram alert if the probe is still failing minutes after boot. That part shipped and closed the incident. But while fixing it I noticed a deeper problem underneath: the probe, and half the launch path around it, shelled out to Docker with no protection at all.

## What the adversarial verifier found in my "complete" fix

My first hardening pass guarded the Docker network check, the helper that decides whether the telemetry network and the egress isolation network exist on the host. I had learned one lesson already: guarding each call site was tried and missed one, so the guarantee belongs at the single place that shells out, inside the helper itself.

I considered the fix complete. My pipeline disagreed. Every fix in this fleet gets a review from a fresh reviewer agent that has no memory of writing the code, and this pass confirmed the fix but refuted the guarantee. The findings were specific, with file and line numbers:

- On a host without the Docker CLI, the launch still died with a raw `FileNotFoundError`, because a `docker inspect` in the container-adoption path runs **before** the guarded network check ever executes.
- Four more shell-outs were unguarded: `docker rm -f` on a stale container, a `Popen` on `docker wait`, the main `docker run`, and the `docker kill` / `docker rm` pair on the timeout path.
- The guarded network check had no `timeout=` at all. A hung Docker daemon would block the launch thread indefinitely, and the blast radius is wider than one task. The daemon runs each launch through `asyncio.to_thread`, which borrows a thread from the event loop's default `ThreadPoolExecutor`. On SIGTERM, the shutdown sequence calls `shutdown_default_executor()`, which joins every worker thread before the process exits. A thread parked inside a `subprocess.run` that has no timeout never returns, so the join never completes: the stuck socket turns a graceful restart into a process that must be killed.

That last one stung. The sibling probe call a few lines away already passed `timeout=60`. The hot-path call simply never got the same treatment, and no test noticed, because tests stub the subprocess boundary.

The author of a fix is the worst person to certify it. I did not find these gaps; a reviewer with no investment in the code being right found them. That review loop is the part of the fleet I trust most now.

## Fail open for telemetry, fail closed for isolation

The core design decision was semantic, not mechanical. One helper answers "does this Docker network exist?", but two very different callers consume the answer.

The **telemetry** caller treats the network as a best-effort feature. If the check cannot run, the launch proceeds without telemetry env. Losing metrics is acceptable; losing the task is not. That is fail open.

The **egress isolation** caller is a security boundary. Isolated tasks run with their network access forced through an allowlist proxy. If the isolation network cannot be confirmed, the task must not run at all, because launching without it would silently drop the sandbox. That is fail closed.

Both callers get a safe answer from the same conservative rule: an unanswerable question resolves to "not present". Telemetry skips itself; isolation refuses to launch. But the refusal message then has to tell the truth, and that forced a second distinction I had initially skipped: "the network is absent" and "the check itself failed" are different facts. Docker reports a genuinely missing network with `No such network` on stderr. Anything else, like `Cannot connect to the Docker daemon`, means the check could not run, and reporting that as "network absent" would send an operator chasing the wrong problem.

The resulting helper is small enough to show in full:

```python
def _docker_network_check(network: str) -> tuple[bool, str]:
    """Return (present, detail) for a docker network. NEVER raises."""
    try:
        proc = subprocess.run(
            ["docker", "network", "inspect", network],
            capture_output=True, text=True, timeout=10)
        if proc.returncode == 0:
            return True, ""
        if "no such network" in proc.stderr.lower():
            return False, "absent"
        return False, "docker check failed"
    except subprocess.TimeoutExpired:
        return False, "docker check failed"
    except Exception:
        return False, "docker check failed"
```

Ten seconds is deliberate. `docker network inspect` answers in milliseconds on a healthy host, so the timeout only ever triggers when the daemon is genuinely stuck, and ten seconds keeps the launch thread's worst case bounded. The heavyweight startup probe keeps its 60-second budget because it runs once, off the hot path. The fail-closed message now carries the reason too: `refusing to run without it (network absent)` versus `(docker check failed)`.

## The adoption path is not allowed to guess

The subtlest change was in container adoption. When the daemon restarts while a task container is still running, the old container survives, and on the next poll the daemon finds it and adopts it: waits for it to finish and uses its result instead of relaunching. That starts with `docker inspect` on the container name.

My first instinct was to treat any inspect failure like the network check: cannot tell, assume absent, launch fresh. The review pass killed that idea with a concrete failure scenario. If the inspect failed because of a transient Docker hiccup while the container **is** still running, "assume absent" clean-launches a second container under the same fixed name, straight into the `docker run --name` conflict this whole function exists to prevent. Two agents on one task, or a launch error, depending on timing.

So the adoption path fails closed on uncertainty. Only Docker's own `no such container` answer counts as confirmed absence; an exception, a timeout, or any other non-zero exit returns a normal failed result with the reason attached:

```python
try:
    state = subprocess.run(
        ["docker", "inspect", "-f", "{{.State.Running}}", name],
        capture_output=True, text=True, timeout=10)
except Exception as e:
    return TaskResult(False, "",
                      f"cannot determine container state for adoption: {e}")
if state.returncode != 0:
    if "no such" in state.stderr.lower():
        return None  # confirmed absent: clean launch
    return TaskResult(False, "",
                      "cannot determine container state for adoption: "
                      f"{state.stderr.strip()}")
```

The caller reads the contract directly: `None` means "nothing to adopt, launch fresh", a `TaskResult` means "stop here and report this", and a successful adoption further down returns the adopted container's own result. Three outcomes, none of them a guess.

The same strictness does not apply to `docker rm -f` on an exited leftover: clearing a dead container is best-effort, so that call just gets a timeout and swallows failures. Every call earned its own policy instead of inheriting a blanket one.

## What a failed launch means now

Every hardened path converges on the same contract: a Docker failure produces a normal `TaskResult` with `ok=False` and a human-readable reason (`docker unavailable: ...`, `cannot determine container state for adoption: ...`, `time limit 30min exceeded`). A failed result flows through the daemon's ordinary verdict path: the failure is recorded against the board task with its reason, and the daemon keeps polling and launches the next task. Nothing unwinds to the top-level loop, and nothing needs a human to restart the process.

The timeout path shows the layering. The container itself runs under a task time limit measured in minutes. When that expires, the daemon kills the container, and now even the cleanup is bounded: `docker kill` and `docker rm -f` each get ten seconds and swallow their own failures, because a cleanup that hangs would be a worse outcome than a leaked container name.

Concretely, the failure modes now map to:

1. Docker CLI missing: the `docker run` Popen raises, the task fails with `docker unavailable`, the daemon lives.
2. Docker daemon hung: control-plane calls cut off at 10 seconds, the task fails with a reason naming the check that failed.
3. Isolation network missing: the task refuses to launch, and the message says whether the network is absent or the check failed.
4. Uninspectable leftover container: the task fails closed rather than risk a double launch.

Honest numbers: in weeks of continuous operation the fleet has hit exactly one of these in the wild, the proxy race that started this story. I have never observed a hung Docker socket on this VM, and the CLI has never gone missing outside tests. That is precisely why the hardening had to come from review rather than from incident pressure. Rare failure modes never generate enough pain to justify themselves reactively; you either fix them while you are already in the file, or you meet them for the first time in production, at night. And the incident that opened this story is closed the same way: the probe now re-checks itself, recovers without a restart, and pages me if it stays broken, so the class of "silently degraded until a human notices" bugs lost both of its known members in one week.

## A test that passes for the wrong reason

One more finding from the review is worth repeating because it generalizes. An existing test verified the egress gate by stubbing `subprocess.run` to return exit code zero, which made the network look present. The gate passed, but only by side effect: the test never asserted the gate consulted the check at all, so a regression that disconnected the gate from the check would still pass. The fix was to stub the named seam directly, the checker function itself, and assert the gate's decision against its return value. A companion test now pins the diagnostic distinction as well: a daemon-down host must produce `docker check failed` in the message and must not claim `network absent`.

Tests that exercise a code path are cheap. Tests that assert the intent of a decision point are the ones that catch wiring regressions.

## Actionable takeaways

- Inventory every shell-out on your agent launch path. Each one is a dependency with its own failure and latency profile, even if it is "just" a local CLI call.
- Put the guard inside the single function that shells out, not at call sites. Call-site guarding was tried here and missed one.
- Give every control-plane subprocess call an explicit timeout. Ten seconds is generous for calls that answer in milliseconds, and it converts a wedged daemon from an outage into one failed task.
- Decide fail-open versus fail-closed per consumer, not per helper. The same unanswerable question should skip optional features and block security boundaries.
- Distinguish "the resource is absent" from "the check failed" in every message an operator will read at 2 a.m.
- Have someone, or something, other than the author try to refute the fix. Every gap described here was found by an adversarial review pass on code I had already judged complete.

## FAQ

### Why does a Docker CLI timeout matter in an AI agent fleet?

Because a hung call blocks more than one task. In this daemon the launch runs in a worker thread, and a stuck Docker socket would have blocked that thread indefinitely and also blocked graceful shutdown on SIGTERM. A 10-second bound turns that into a single failed launch with a readable reason.

### What is the difference between fail open and fail closed here?

Fail open: if the telemetry network cannot be confirmed, the task launches anyway, just without metrics. Fail closed: if the egress isolation network cannot be confirmed, the task does not launch at all, because running without the sandbox would be a silent security downgrade. Both consume the same check; the policy lives with the consumer.

### Why not treat an uninspectable container as absent and relaunch?

Because the container may still be running. Guessing "absent" would start a second container under the same name and collide with the live one. Absence has to be confirmed by Docker's own no-such-container answer; everything else fails closed with a reason.

### Should every external tool call be hardened this way?

Every one on a critical path, yes. The pattern is small: wrap the subprocess boundary, pass an explicit timeout, classify the failure into a structured state, and let the orchestration loop keep running. The judgment call is the per-consumer fail-open versus fail-closed policy, and that is exactly the part a generic wrapper cannot decide for you.

## Related reading

- [Optimizing Agent Role Prompts Offline: What SkillOpt Automates, and the Manual Loop My Fleet Already Runs](https://www.tytarenkoagency.com/blog/optimizing-agent-role-prompts-offline-what-skillopt-automates-and-the-manual-loop-my-fleet-already-runs)
- [Building Robust AI Systems: Handling Orphan Tasks and Cross-Vendor Authentication](https://www.tytarenkoagency.com/blog/building-robust-ai-systems-orphan-tasks-authentication)
- [Graph engineering is easy. Not blowing your budget on one workflow is the hard part](https://www.tytarenkoagency.com/blog/graph-engineering-is-easy-not-blowing-your-budget-on-one-workflow-is-the-hard-part)