Resilience Patterns (Circuit Breaker, Bulkhead, Retry)

In a distributed system, dependencies will fail or slow down. Resilience patterns contain that failure so one sick component does not cascade into a system-wide outage. The recurring enemy is the cascading failure: a slow dependency exhausts callers' threads/connections, which makes them slow, which takes down their callers.

The patterns below are composable defenses placed at the call site to a dependency. Most are now provided by the mesh sidecar or a resilience library rather than hand-rolled. Source: Nygard, Release It!

The core patterns

• Timeout — never wait unboundedly. An unbounded call is the root of most cascades; every remote call needs a deadline. The simplest and most important pattern.
• Retry with backoff + jitter — retry transient failures, but with exponential backoff and randomized jitter so clients do not synchronize into a retry storm that hammers a recovering service. Only retry idempotent operations (see Idempotency and Exactly-Once Processing) and cap attempts with a retry budget.
• Circuit breaker — track failures to a dependency; when they exceed a threshold, open the circuit and fail fast (skip the call) for a cooldown, then allow a few half-open trial calls to test recovery before closing again. Stops callers from waiting on a known-dead dependency.
• Bulkhead — isolate resources per dependency (separate thread/connection pools) so one saturated dependency cannot consume all capacity — named after a ship's watertight compartments.
• Load shedding — under overload, reject low-priority work early (related to Rate Limiting Architecture) to protect the core.
• Fallback — degrade gracefully (cached/default response) when a dependency is down — but only where a degraded answer is genuinely correct; see Kevin's No-Fallbacks Policy for when a silent fallback is worse than a loud failure.

Circuit breaker states

Design implications

• Timeouts are non-negotiable. Set them at every layer; a missing timeout deep in the stack is a latent outage.
• Retries amplify load. Naive retries turn a partial outage into a total one; backoff + jitter + budgets are mandatory, and retries must be idempotent.
• Tune breakers to the dependency. Thresholds and cooldowns differ per dependency; too sensitive trips on noise, too lax defeats the purpose.
• Resilience needs observability. You cannot tune breakers/timeouts without metrics on failure rates and latency (Observability Architecture).
• Fail loud where correctness matters. A fallback that hides a real failure (e.g. returning empty data as if valid) violates No-Fallbacks Policy.

Agent-system translation

For agent harnesses, use Agent Resilience Patterns. The same ideas apply, but the dependency is often a model, MCP server, browser, workspace, account permission, or human approval queue rather than a normal service. The practical translation is: circuit breakers become kill switches, bulkheads become tool/workspace blast-radius limits, retries need idempotency keys, and distributed traces become action traces with model and tool hops.

---

Timeline

• 2026-06-30 | Linked the classic distributed-systems catalog to Agent Resilience Patterns after X bookmark artifact review surfaced a direct pattern mapping for agent harnesses. Source: X/@divaagurlxw, 2026-06-24
• 2026-06-02 | Page created. Captured cascading failure as the core threat, the timeout/retry-with-backoff/circuit-breaker/bulkhead/load-shed/fallback catalog, the breaker state machine, and implications (timeouts everywhere, retry amplification, per-dependency tuning, fail-loud). Source: Nygard, Release It!