Agent Resilience Patterns
Agent systems inherit the same failure modes as distributed systems: retries duplicate side effects, shared tools create blast radius, queues overload, and healthy-looking runs can still be wrong. The durable solution is to translate distributed-systems resilience patterns into harness-level controls.
The root bookmark argues that serious AI engineers should learn harness engineering, context engineering, caching tradeoffs, KV-cache pressure, prefill/decode latency, tool-protocol costs, evals, and observability instead of stopping at prompt engineering. The linked source thread makes the sharper architecture move: concepts learned for distributed systems have direct agentic-AI equivalents. Source: X/@divaagurlxw, 2026-06-24; Source: X/@divaagurlxw, 2026-06-06
Pattern Mapping
| Distributed-systems pattern | Agent-system equivalent | Harness question |
|---|---|---|
| Circuit breaker | Kill switch after repeated failure | When should the agent stop retrying and escalate? |
| Bulkhead | Blast-radius limiter per tool, workspace, account, or permission scope | What can one bad run damage? |
| Saga pattern | Compensation chain for multi-step tool workflows | If step 4 fails, which prior side effects need explicit repair? |
| Service discovery | MCP/tool registry with capability, auth, and health metadata | Which tools exist, who owns them, and are they safe to call now? |
| Idempotency key | Safe retry key for side-effecting actions | Can a retry create duplicate emails, tickets, charges, commits, or deploys? |
| Dead-letter queue | Escalation queue for stuck or low-confidence tasks | Where do failed runs go so they are not silently lost? |
| Health check | Confidence probe, eval, smoke test, or environment probe | Is the model/tool/workspace actually ready for this task? |
| Rate limiter | Token, tool-call, cost, and concurrency budget per agent | What prevents a loop from turning into an invoice or outage? |
| API gateway | LLM gateway and tool gateway | Where do auth, logging, routing, model choice, quotas, and policy live? |
| Session store | Agent context checkpoint | What state survives compaction, process death, or human handoff? |
| Leader election | Orchestrator/owner selection across agents | Who is allowed to decide the next step when many agents can act? |
| Backpressure | Queue-depth and budget pressure surfaced to planner | When should intake slow, split, shed, or ask a human? |
| Canary deploy | Shadow-mode or preview-mode agent testing | Can this agent run without writing to production first? |
| Distributed trace | Action trace with model/tool/workspace hops | Can a reviewer reconstruct why an action happened? |
Design Implications
- Reliability belongs in the harness. A more capable model can still loop, duplicate side effects, or call the wrong tool. The harness owns budgets, circuit breakers, idempotency, queueing, escalation, and audit traces.
- Tools need dependency contracts. Treat tools like remote services: timeouts, health, auth, owner, rate limits, retry policy, and failure semantics should be explicit before agents call them.
- Agent autonomy is a tradeoff surface. The CAP-style triangle for agents is autonomy, reliability, and oversight. Maximizing all three is not free; a product chooses where humans, budgets, and evals sit.
- Durability is resilience. Checkpoints, resumable sessions, and event logs turn failures into restartable workflows. Without them, a long-running agent is a fragile process with expensive context.
Stack Placement
Use this page with Harness Engineering when designing agent systems, with Resilience Patterns (Circuit Breaker, Bulkhead, Retry) when mapping classic failure controls, and with Durable Agent Execution Architecture when a run must survive crashes, approvals, retries, or deploys. The operational target is boring: every repeated failure mode should become a harness invariant, not a prompt reminder.
Timeline
- 2026-06-30 | Page created from the deep review of the @divaagurlxw AI-engineering stack bookmark and its linked distributed-systems mapping thread. Promoted because the source is not just another harness checklist; it gives a concrete reliability vocabulary for agent control planes. Source: X/@divaagurlxw, 2026-06-24