Resilience Patterns

How Distributed Systems Fail

Distributed systems do not fail cleanly. In a monolith, a crash is a crash: the process dies and the error is immediate. In a distributed system, failures are partial, probabilistic, and often invisible. A service can be running while returning garbage. A database can be up while timing out on every write. A network can be connected while dropping 10% of packets.

Resilience engineering is about designing for failures that you cannot prevent. Hardware fails, networks partition, dependencies time out, bugs are deployed. The goal is not to eliminate these events, it is to bound their impact: contain the failure domain, degrade gracefully under load, recover automatically without human intervention.

Failure taxonomy

The fallacies of distributed computing

Peter Deutsch and James Gosling catalogued the eight assumptions engineers make about networks that are always false in production. Each false assumption is a category of failure that resilience patterns must address.

Retries and Timeouts

Timeouts and retries are the first line of resilience: every outbound call must have a timeout, and transient failures should be retried with backoff. Without timeouts, a slow dependency occupies a thread forever. Without retries, a single dropped packet causes a user-facing error. Without backoff, retries amplify load on an already struggling service.

Timeout types

Retry strategies

Only retry idempotent operations. A GET request can always be retried. A POST that creates a record must not be retried without an idempotency key, you risk creating the record twice. An idempotency key (a UUID in a request header) lets the server detect and discard duplicate requests, making any operation safe to retry.

import asyncio, random, uuid, httpx

async def call_with_retry(
    url: str,
    payload: dict,
    max_attempts: int = 3,
    base_delay: float = 0.5,
) -> dict:
    idempotency_key = str(uuid.uuid4())

    for attempt in range(max_attempts):
        try:
            async with httpx.AsyncClient(timeout=3.0) as client:
                resp = await client.post(
                    url,
                    json=payload,
                    headers={"Idempotency-Key": idempotency_key},
                )
                resp.raise_for_status()
                return resp.json()

        except (httpx.TimeoutException, httpx.HTTPStatusError) as e:
            if attempt == max_attempts - 1:
                raise
            # Exponential backoff with full jitter
            max_wait = base_delay * (2 ** attempt)
            wait = random.uniform(0, max_wait)
            await asyncio.sleep(wait)

    raise RuntimeError("unreachable")

What NOT to retry

Circuit Breaker

The circuit breaker pattern prevents cascading failures by stopping calls to a service that is known to be failing. Named after the electrical breaker that interrupts current when a fault is detected, it implements a state machine with three states: Closed (normal operation), Open (fail-fast, no calls), and Half-Open (probe to test recovery).

The key insight is fail-fast under failure. Without a circuit breaker, a slow service causes the caller to queue up waiting threads until its thread pool exhausts. With a circuit breaker open, the caller returns an error immediately without waiting. This prevents the failure from propagating upstream and frees threads to serve other requests.

The three states

Implementation with a fallback

from circuitbreaker import circuit

@circuit(
    failure_threshold=5,     # trip after 5 consecutive failures
    recovery_timeout=30,     # wait 30s before half-open probe
    expected_exception=Exception,
)
async def get_recommendations(user_id: str) -> list[str]:
    return await recommendations_service.get(user_id)

async def handler(user_id: str):
    try:
        recs = await get_recommendations(user_id)
        return recs
    except CircuitBreakerOpen:
        # Circuit is open, return cached or static fallback
        # Do NOT let this error propagate to the user
        return get_fallback_recommendations()
    except Exception as e:
        # Service error, circuit will count this failure
        log.warning("recommendations failed", error=str(e))
        return get_fallback_recommendations()

def get_fallback_recommendations() -> list[str]:
    # Return popular items, recently viewed, or empty list
    # Never fail the entire request because of a non-critical service
    return popular_items_cache.get() or []

What triggers a circuit breaker

Bulkheads, Load Shedding, and Graceful Degradation

Bulkhead pattern

A bulkhead is a partition in a ship's hull that isolates flooding to one compartment. In software, it is resource isolation: separate thread pools, connection pools, or process limits for different dependencies so that one failing dependency cannot exhaust resources shared with healthy ones.

Without bulkheads, a slow payments service consumes all available threads in the application's shared thread pool. New requests for any endpoint, including the fast, healthy ones, queue and eventually time out. The cascade is horizontal: one failing dependency brings down everything.

from concurrent.futures import ThreadPoolExecutor

# Isolated pools per dependency, one pool exhausting
# does not affect the others
POOLS = {
    "payments":        ThreadPoolExecutor(max_workers=10),
    "recommendations": ThreadPoolExecutor(max_workers=5),
    "notifications":   ThreadPoolExecutor(max_workers=3),
}

async def call_payments(payload):
    loop = asyncio.get_event_loop()
    try:
        return await asyncio.wait_for(
            loop.run_in_executor(POOLS["payments"], _call_payments, payload),
            timeout=2.0,   # per-pool timeout
        )
    except asyncio.TimeoutError:
        raise PaymentsUnavailableError()

# If recommendations pool is exhausted (5 threads all waiting),
# payments pool (10 threads) is completely unaffected

Load shedding

Load shedding is the deliberate rejection of requests when the system is operating above its sustainable capacity. The alternative, accepting all requests and attempting to serve them all, leads to queuing, growing latency, and eventual collapse. A system that rejects 20% of requests under overload can serve the remaining 80% well. A system that accepts all requests serves 100% of them badly.

Graceful degradation

A resilient system distinguishes between critical and non-critical features. When a non-critical dependency fails, the system continues operating at reduced capability rather than returning an error. When a critical dependency fails, it surfaces the error clearly rather than silently returning wrong data.

Health checks and readiness