Failure & Fault-Tolerance Theory: The Complete System Design Reference

A self-contained reference for treating failure as a first-class design input, not an afterthought. At scale, something is always broken — disks, nodes, networks, whole datacenters. The engineers who build systems that survive don’t hope for the best; they reason about failure models, redundancy math, and blast-radius containment explicitly. This chapter is that discipline.

How to use this: Part 1 is the reference card. Part 2 maps the territory. Part 3 is the full depth with pros/cons per technique. Part 4 is exhaustive interview prep with counter-question ladders.

Key takeaways

• You cannot prevent failures; you bound their probability (redundancy) and their blast radius (isolation and degradation).
• Serial dependencies multiply availability down; parallel redundancy multiplies failure probability away — but only if failures are independent.
• Reducing MTTR is often a cheaper path to more nines than increasing MTBF.
• Retries without budgets, backoff, and jitter cause retry storms and metastable failures that persist after the trigger is gone.

PART 1 — CHEATSHEET (Reference Card)

Every concept in this document, condensed.

The one idea

You cannot prevent failures; you can only bound their probability and their blast radius. Reliability engineering = choosing redundancy to make failure rare, and isolation + graceful degradation to make the failures that slip through small and survivable.

Core vocabulary (one line each)

• Fault → error → failure — a defect (fault) causes a bad state (error) that may cause deviation from spec (failure).
• Failure models — crash-stop, crash-recovery, omission, timing, Byzantine (arbitrary/malicious).
• Failure detector — module that suspects crashed nodes; perfect detection is impossible in async (FLP).
• Phi-accrual detector — outputs a suspicion level, not a boolean — tunable, adaptive.
• Byzantine Generals — agreement with traitors; needs 3f+1 nodes to tolerate f liars.
• Redundancy — extra components so failures don’t cause outage (N-modular, replication).
• MTBF / MTTR — mean time between failures / mean time to recovery; availability = MTBF/(MTBF+MTTR).
• Nines — 99.9% ≈ 8.76 h/yr down; 99.99% ≈ 52.6 min; 99.999% ≈ 5.26 min.
• Bulkhead — isolate resources so one overload can’t sink the whole ship.
• Circuit breaker — stop calling a failing dependency; fail fast, allow recovery.
• Timeout / retry / backoff / jitter — bound waits / re-attempt / spread re-attempts / randomize them.
• Idempotency — safe to apply twice (makes retries safe).
• Load shedding / backpressure — drop or reject excess work / signal upstream to slow down.
• Graceful degradation — reduce functionality instead of failing entirely.
• Fail-fast / crash-only — surface errors immediately / make restart the only recovery path.
• Cascading / metastable failure — failure that propagates / a bad stable state that persists after the trigger is gone.
• Chaos engineering — inject failures in production to validate resilience.
• SLI / SLO / error budget — measured indicator / target / allowed unreliability to spend.

Failure models (increasing difficulty)

Availability nines (memorize)

Redundancy math

• Availability A = MTBF / (MTBF + MTTR) — reducing MTTR is often easier than raising MTBF.
• Serial (dependency chain): A_total = A₁ × A₂ × … (more dependencies → lower availability).
• Parallel (redundant): unavailability = (1−A₁) × (1−A₂) × … (redundancy multiplies the failure probabilities → much higher availability).
• Tolerate f crash faults → 2f+1 replicas; f Byzantine → 3f+1.

Quick decision rules

• Calls to any remote dependency → timeout + bounded retry (backoff+jitter) + circuit breaker.
• Shared resource pools → bulkheads (separate pools per tenant/dependency).
• Overload → load shedding + backpressure, never unbounded queues.
• Must hit high availability → redundancy (parallel) + low MTTR (fast detection/failover).
• Validate resilience → chaos experiments with a hypothesis and blast-radius limit.

Top gotchas (litmus tests)

1. Retries amplify outages (retry storms) — always budget + exponential backoff + jitter.
2. Perfect failure detection is impossible in async systems (FLP) — you only ever suspect.
3. Lowering MTTR beats raising MTBF — fast recovery often gives more nines, cheaper.
4. Adding dependencies multiplies failure — serial availability drops with each one.
5. Metastable failures persist after the trigger is gone — recovery needs to break the loop (shed load), not just remove the cause.
6. Timeouts that are too long turn a slow dependency into a thread-pool exhaustion outage.
7. Health checks that only ping /health miss real dependency failures (shallow vs deep checks).
8. Idempotency is a prerequisite for safe retries — without it, retries double-apply.
9. Byzantine tolerance needs 3f+1, not 2f+1 — majority doesn’t stop lies.
10. “It failed over” is not enough — measure data loss (RPO) and recovery time (RTO).

PART 2 — OUTLINE (full map)

1. Fault → error → failure and the dependability taxonomy
2. Failure models
3. Failure detectors
4. The Byzantine Generals problem and the 3f+1 bound
5. Redundancy and the math of availability
6. Fault isolation: bulkheads and cells
7. Resilience patterns: timeouts, retries, circuit breakers, load shedding, backpressure, degradation
8. Fail-fast and crash-only software
9. Cascading and metastable failures
10. Chaos engineering
11. Redundancy topologies (active-active, active-passive, standby)
12. SRE: SLIs, SLOs, and error budgets
13. Decision guide
14. Make it stick — the teaching tutorial (availability math, cascade & metastable pictures, flashcards)

PART 3 — DEEP DIVE

1. Fault → error → failure and the dependability taxonomy

The dependability taxonomy (Avizienis, Laprie, Randell, Landwehr, 2004) gives precise words: a fault is the root defect (a bug, a bad disk); an error is the resulting incorrect internal state; a failure is when the service deviates from its specification (the user sees it). The chain is fault → error → failure, and resilience means breaking it: prevent faults, tolerate errors (so they don’t become failures), and contain failures. The attributes of dependability — availability, reliability, safety, integrity, maintainability — are what you’re trading. Being precise about which you’re optimizing (and that a fault isn’t yet a failure) is a senior signal.

2. Failure models

You must state your failure model before you can claim correctness:

• Crash-stop (fail-stop): a node halts and never returns. Simplest; tolerated by majority quorums.
• Crash-recovery: a node crashes and may rejoin with stale state — needs stable storage to recover safely.
• Omission: a node or link drops some messages (send/receive omission) — handled with timeouts and retransmission.
• Timing: responses violate timing assumptions (too slow) — relevant in real-time/partially-synchronous systems.
• Byzantine: a node behaves arbitrarily — buggy, compromised, or malicious; may send conflicting information to different peers. Hardest; needs 3f+1 and cryptographic techniques.

Each model is strictly harder than the last; choosing too weak a model (assuming crash-only when nodes can corrupt) is a real source of outages.

3. Failure detectors

A failure detector is the module that decides “is that node dead?” In a purely asynchronous system you can’t distinguish a crashed node from a slow one — so perfect detection is impossible (this is the practical face of FLP: Fischer, Lynch, Paterson, 1985, showed consensus can’t be guaranteed in async with one crash, precisely because you can’t reliably detect it). Chandra & Toueg (1996) formalized failure detectors by their completeness (every crashed node is eventually suspected) and accuracy (correct nodes aren’t wrongly suspected), and proved that an eventually-strong (◊S) detector is the weakest oracle sufficient to solve consensus.

In practice, detection is timeout-based and a tradeoff: aggressive timeouts catch failures fast but cause false positives (and flapping); conservative ones are stable but slow. The phi-accrual detector (Hayashibara et al., 2004) improves this by outputting a continuous suspicion level (φ) based on the statistical distribution of recent heartbeat arrivals, letting different actions trigger at different confidence thresholds — adaptive to network conditions.

Pros/cons: you trade detection speed against false-positive rate; there is no “correct” timeout, only a chosen point on that curve.

4. The Byzantine Generals problem and the 3f+1 bound

Lamport, Shostak, and Pease (1982) framed the problem: generals must agree on attack/retreat, but some are traitors sending conflicting messages. The result: to tolerate f Byzantine faults you need 3f+1 participants (with unsigned messages), and quorums of 2f+1. Intuition: any two 2f+1 quorums intersect in at least f+1 nodes, guaranteeing ≥1 honest node in common — so honest nodes can’t be split into disagreeing majorities. With cryptographic signatures the bounds relax somewhat, but the canonical figure to know is 3f+1. This is why crash-tolerant consensus (2f+1) is cheaper and used inside trusted datacenters, while Byzantine tolerance is reserved for adversarial / multi-organization / blockchain settings.

5. Redundancy and the math of availability

Availability A = uptime fraction = MTBF / (MTBF + MTTR). Two levers: increase MTBF (more reliable components — expensive, diminishing returns) or decrease MTTR (faster detection + recovery — often the cheaper path to more nines). This is why fast failover, good monitoring, and automated remediation matter so much: halving recovery time is as good as doubling reliability.

Composition:

• Serial dependencies (A needs B needs C): A_total = A_A × A_B × A_C. Every dependency you add lowers availability — a service depending on five 99.9% services is at best 99.5%. This is the quiet killer of microservice availability.
• Parallel redundancy (N replicas, system up if any is up): unavailability = ∏(1−Aᵢ). Two 99% replicas → unavailability 0.01×0.01 = 0.0001 → 99.99%. Redundancy multiplies failure probabilities, which is hugely powerful — if failures are independent (correlated failures, e.g. same rack/AZ/bad deploy, break this).

N-modular redundancy (NMR): run N copies and vote (e.g. triple modular redundancy, TMR) — masks faults (including some Byzantine) at the cost of N× resources. Common in aerospace.

Key caveat: the math assumes independent failures. Real outages are often correlated (a bad config push, a shared dependency, an AZ power loss) — which is why isolation and diversity matter as much as raw replica count.

6. Fault isolation: bulkheads and cells

Borrowed from ships, a bulkhead partitions resources so a flood in one compartment can’t sink the vessel: separate thread pools / connection pools / queues per dependency or per tenant, so one slow dependency or noisy tenant can’t exhaust the shared resource and take down everything. Cell-based architecture generalizes this: partition the whole system into independent cells (each a full stack serving a subset of users), so a failure is contained to one cell’s users — bounding the blast radius. Shuffle sharding further reduces the chance any two tenants share all the same cells.

Pros: failures stay local; predictable blast radius; supports incremental deploys (one cell at a time). Cons: more operational complexity and some resource overhead (less pooling/sharing).

7. Resilience patterns

The practitioner’s toolkit (Nygard, Release It!, 2007):

• Timeouts: never wait unbounded on a remote call — an un-timed-out call holds a thread/connection; enough of them exhaust the pool and convert a slow dependency into a total outage. Set timeouts deliberately (often based on the dependency’s p99).
• Retries with exponential backoff + jitter: retry transient failures, but bound them (retry budget), back off exponentially, and jitter to avoid synchronized retry waves. Retries without these amplify outages (retry storms).
• Idempotency: the prerequisite for safe retries — use idempotency keys so a duplicated request doesn’t double-apply.
• Circuit breaker: track failures to a dependency; when they exceed a threshold, open the circuit (fail fast without calling) for a cooldown, then half-open to test recovery. Prevents hammering a down service and gives it room to recover.
• Load shedding: under overload, reject excess work early (return 429/503 fast) rather than queueing it — queued work just adds latency and eventually OOMs.
• Backpressure: propagate “slow down” upstream (bounded queues, credit/flow control) instead of absorbing unbounded load.
• Graceful degradation: serve reduced functionality (cached/stale data, fewer features) instead of total failure — e.g. show the catalog without personalized recommendations if the rec service is down.

Pros: dramatically shrink blast radius and recovery time. Cons: each adds complexity and tuning; misconfigured (too-long timeouts, too-aggressive retries) they cause the outages they’re meant to prevent.

8. Fail-fast and crash-only software

• Fail-fast: detect problems early and surface them immediately (validate inputs, check preconditions, short timeouts) rather than limping along into a worse state.
• Crash-only software (Candea & Fox, 2003): design components so the only way they stop is to crash, and the only way they start is to recover — i.e. recovery code is the normal path, exercised constantly, so it actually works when needed. This eliminates fragile, rarely-tested “graceful shutdown” paths and makes restart a reliable, fast recovery mechanism. Microservices/containers that are killed and rescheduled embody this.

Pros: simpler, well-tested recovery; fast restart; no half-dead states. Cons: requires idempotent startup and externalized state (so a crash doesn’t lose work).

9. Cascading and metastable failures

• Cascading failure: one component’s failure overloads others (e.g. a dependency slows, callers’ threads pile up, they fail, their callers fail) — failure propagates through the dependency graph. Prevented by timeouts, circuit breakers, bulkheads, and load shedding.
• Metastable failure (Bronson et al., 2021): a system enters a stable bad state that persists even after the original trigger is removed, because a feedback loop sustains it. Classic example: a brief overload causes retries; the retries become the new load, keeping the system overloaded indefinitely. Removing the trigger doesn’t help — you must break the loop (shed load, disable retries, drain queues) to escape. Recognizing that “the cause is gone but it’s still down” indicates a metastable state is a strong interview signal.

10. Chaos engineering

Chaos engineering (formalized at Netflix; Basiri et al., 2016) is the practice of deliberately injecting failures — killing instances, adding latency, partitioning networks — to validate that resilience mechanisms actually work, ideally in production. The discipline: form a hypothesis about steady-state behavior, inject a real-world failure, limit the blast radius, and verify the system holds (or learn that it doesn’t). The principle: you don’t know your failover works until you’ve made it fail. The Netflix “Chaos Monkey”/Simian Army popularized routine instance termination.

Pros: finds latent weaknesses (untested failover, hidden dependencies) before customers do; builds confidence. Cons: requires maturity, good observability, and careful blast-radius control; reckless chaos causes real incidents.

11. Redundancy topologies

• Active-active: all replicas serve traffic; failure of one just removes capacity (and requires the others to absorb it — capacity planning must assume N−1). Best availability, but needs conflict handling if they accept writes.
• Active-passive (hot standby): a standby replica stays in sync and takes over on failover — simpler (no concurrent-write conflicts) but the standby’s capacity is idle, and failover has a gap (RTO).
• Warm / cold standby: standby partially ready / must be provisioned on demand — cheaper, slower recovery.
• Disaster recovery metrics: RTO (recovery time objective — how long to recover) and RPO (recovery point objective — how much data loss is acceptable). State both; “we fail over” without RTO/RPO is hand-waving.

12. SRE: SLIs, SLOs, and error budgets

Google’s SRE framing (Beyer et al., 2016): an SLI (Service Level Indicator) is a measured quantity (e.g. % of requests served < 200 ms, success rate); an SLO (Objective) is the target (e.g. 99.9% of requests succeed); an error budget is the allowed unreliability (1 − SLO = the budget you can “spend” on risk/velocity). The power move: error budgets align reliability and feature velocity — if you’re within budget, ship fast; if you’ve burned it, freeze risky changes and invest in reliability. It turns “how reliable should we be?” from an argument into a number, and explicitly rejects chasing 100% (uneconomical and unnecessary — users can’t tell beyond a point).

13. Decision guide

Every remote call:
   ► TIMEOUT (≈ dependency p99) + BOUNDED RETRY (backoff+jitter, idempotent only) + CIRCUIT BREAKER

Overload:
   ► LOAD SHEDDING (reject early) + BACKPRESSURE (bounded queues) + GRACEFUL DEGRADATION
   ► If still down after trigger removed → suspect METASTABLE: break the loop (kill retries, drain)

Isolation:
   ► BULKHEADS (per-dependency/tenant pools) + CELLS (bound blast radius)

Availability target:
   ► REDUNDANCY (parallel, independent failure domains) + LOW MTTR (fast detect/failover)
   ► Need f-crash tolerance → 2f+1; f-Byzantine → 3f+1
   ► Set SLOs + ERROR BUDGET; don't chase 100%

Validate:
   ► CHAOS experiments (hypothesis + limited blast radius)

Reach-for / avoid:

• Retries — for: transient faults on idempotent ops. Avoid when: unbounded or non-idempotent (storms / double-apply).
• Circuit breaker — for: protecting against a failing dependency. Avoid when: the dependency is critical and you have no degraded mode (then you just fail fast — design a fallback).
• Active-active — for: max availability. Avoid when: concurrent writes conflict and you can’t reconcile (use active-passive).
• Chaos engineering — for: mature systems with observability. Avoid when: you lack blast-radius controls.

PART 4 — INTERVIEW ARSENAL

How to wield this. Senior signals: (1) you state the failure model and RTO/RPO, not just “it fails over”; (2) you reach for timeouts + bounded retries + circuit breakers + bulkheads reflexively and warn about retry storms / metastable failure; (3) you reason with availability math (serial multiplies down, parallel multiplies failure away; MTTR vs MTBF). Each question has a model answer and counter-ladder.

A. Fundamentals

Q1. Distinguish fault, error, and failure, and why it matters. Answer: A fault is the defect (bug, bad disk), an error is the resulting bad internal state, a failure is user-visible deviation from spec. It matters because resilience is about breaking the chain — tolerating errors so they never become failures (redundancy, isolation), not just preventing faults. A fault that’s masked never becomes a failure. Counter-ladder:

• “Give an example of an error that isn’t a failure.” → A bit flip corrected by ECC, or a crashed replica masked by a healthy one.
• “Which dependability attribute are you optimizing?” → Name it (availability vs integrity vs safety) — they trade off.

Q2. What failure models are there, and which does majority consensus tolerate? Answer: Crash-stop, crash-recovery, omission, timing, Byzantine — increasing difficulty. Majority quorums (2f+1) tolerate f crash faults; Byzantine needs 3f+1. Choosing too weak a model (assuming crash-only when corruption/malice is possible) is a real source of outages. Counter-ladder:

• “Why 3f+1 for Byzantine?” → Quorums of 2f+1 intersect in ≥ f+1 nodes, guaranteeing a shared honest node so liars can’t split the agreement.
• “Do you need Byzantine tolerance inside your datacenter?” → Usually no (trusted nodes) — crash tolerance suffices; BFT for adversarial/multi-org/blockchain.

B. Detection & availability math

Q3. Why can’t you perfectly detect a failed node, and what do you do instead? Answer: In an asynchronous network you can’t distinguish a crashed node from a slow one (FLP) — any detector is timeout-based and trades speed against false positives. So you suspect rather than know; use adaptive detectors (phi-accrual outputs a suspicion level) and design protocols that stay safe under false suspicion and only lose liveness temporarily. Counter-ladder:

• “Aggressive vs conservative timeouts?” → Aggressive = fast detection but flapping/false positives; conservative = stable but slow recovery.
• “What’s the weakest detector that solves consensus?” → ◊S (eventually-strong), per Chandra–Toueg.

Q4. You need four nines. Walk the availability math. Answer: A = MTBF/(MTBF+MTTR). Two levers: raise MTBF or cut MTTR — cutting MTTR (fast detection + automated failover) is usually cheaper per nine. Use parallel redundancy across independent failure domains: two 99% replicas give 99.99% if failures are independent. Watch serial dependencies — each one multiplies availability down, so five 99.9% dependencies cap you around 99.5%. Counter-ladder:

• “Why doesn’t adding replicas always help?” → Correlated failures (same AZ, bad deploy, shared dependency) violate independence — diversify failure domains.
• “Three nines downtime per year?” → ~8.76 hours. Four nines ~52.6 min. Five nines ~5.26 min.
• “Cheapest way to add a nine?” → Often reduce MTTR (faster rollback/failover/detection), not buy more reliable hardware.

C. Resilience patterns

Q5. A downstream dependency slows to 5s responses and your whole service goes down. Explain and fix. Answer: Threads/connections block on the slow call (no/too-long timeout), the pool exhausts, and unrelated requests can’t be served — a cascading failure from resource exhaustion. Fixes: short timeouts (≈ dependency p99), a circuit breaker to fail fast when it’s unhealthy, bulkheads (separate pool for that dependency so it can’t starve others), and graceful degradation (serve cached/degraded results). Counter-ladder:

• “Why do timeouts alone not fully fix it?” → Under sustained failure you still hammer the dependency; add a circuit breaker to stop calling and let it recover.
• “How do bulkheads help here?” → They cap how many threads that dependency can consume, so it can’t exhaust the whole pool.

Q6. After an incident the trigger is resolved but the system stays down. What’s happening? Answer: A metastable failure — a feedback loop (commonly retries that became the load) holds the system in a stable bad state independent of the original trigger. You must break the loop: shed load, disable/limit retries, drain queues, then bring traffic back gradually. Removing the original cause isn’t enough. Counter-ladder:

• “How prevent it?” → Retry budgets + backoff + jitter, load shedding, and bounded queues so overload can’t self-sustain.
• “How recover safely?” → Reduce load below the recovery threshold, then ramp up — not flip everything back at once.

D. Architecture & process

Q7. Design the resilience strategy for an order service depending on payment, inventory, and notifications. Answer: Classify dependencies by criticality: payment + inventory are critical (order can’t complete without them) — wrap in timeouts + bounded retries (idempotency keys) + circuit breakers, and define a degraded mode (queue the order as “pending” if payment is briefly down). Notifications are non-critical — make them async/best-effort so their failure never blocks an order (graceful degradation). Bulkhead the pools so notifications can’t starve payment. Watch serial-availability: minimize hard dependencies on the critical path. Counter-ladder:

• “Payment retry — risk?” → Double-charge; require idempotency keys so retries are safe.
• “How keep notification failures from hurting availability?” → Move them off the synchronous path (async queue); their outage doesn’t affect order success.

Q8. What’s an error budget and how does it change behavior? Answer: Error budget = 1 − SLO; the allowed unreliability. It aligns reliability with velocity: within budget → ship features fast; budget exhausted → freeze risky changes and invest in reliability. It also rejects chasing 100% (uneconomic, and users can’t perceive it). It turns reliability decisions into a shared number rather than an argument. Counter-ladder:

• “Who ‘owns’ the budget?” → Shared between dev and SRE; burning it has agreed consequences (change freeze).
• “Why not aim for 100%?” → Cost explodes near 100%, and added nines deliver no perceptible user benefit past the dependency floor.

E. Worked drill — driving a design end-to-end

Watch failure modeling, blast-radius containment, and explicit RTO/RPO drive the design.

Prompt: “Design for resilience: a payment-processing API that must be highly available (target four nines), must never double-charge, and must survive the loss of an entire availability zone.”

1 — State the failure model and targets. “Trusted infrastructure → crash/omission model (not Byzantine), so crash-tolerant consensus suffices. Targets: 99.99% availability (~52 min/yr), RPO = 0 for committed payments (no lost charges), small RTO (fast failover). Never double-charge → idempotency is non-negotiable.”

2 — Redundancy across independent failure domains. “Survive an AZ loss → deploy across ≥3 AZs, with the payment ledger on a consensus-replicated store (Raft/Paxos, 2f+1 = at least 3 nodes spread across AZs) so a majority survives one AZ’s loss with RPO 0 (committed = on a majority). Stateless API tier runs active-active across AZs; losing one AZ just removes capacity, so I provision for N−1 (each AZ can carry the load of a failed peer).”

3 — Idempotency (never double-charge). “Every charge carries a client-supplied idempotency key; the ledger records processed keys, so retries, failovers, and at-least-once delivery can’t double-apply. This makes retries safe, which is what lets me retry aggressively elsewhere.”

4 — Dependency resilience. “Calls to the card network (external, can be slow/down): timeout at its p99, bounded retries with backoff+jitter (idempotent via the key), and a circuit breaker so a card-network brownout doesn’t exhaust our threads (cascading failure). Bulkhead the card-network pool from internal calls. If the network is down, degrade: accept and queue the payment as ‘authorizing’ rather than hard-failing, reconciling when it recovers.”

5 — Containment. “Cell-based: partition merchants across cells so a bad deploy or poisoned state hits one cell’s merchants, not all — bounding blast radius. Deploys roll cell-by-cell behind the SLO. Load shedding (429) protects against surge rather than unbounded queuing (which risks a metastable retry spiral).”

6 — MTTR and validation. “Four nines is reached as much by low MTTR as redundancy: automated AZ-failure detection + failover, health checks that are deep (verify ledger reachability, not just process liveness), and one-click rollback. Validate it all with chaos experiments — kill an AZ in a controlled window and confirm RPO 0 / RTO target hold, with a hypothesis and bounded blast radius.”

7 — Tradeoffs stated. “I bought availability with cross-AZ consensus (paying inter-AZ write latency for RPO 0) and active-active stateless tiers (paying N−1 over-provisioning), and bought safety with idempotency keys everywhere. I contained blast radius with cells and bulkheads at the cost of operational complexity. I explicitly did not pursue five nines (cost vs. benefit) and set an error budget at 99.99% to govern release velocity. Byzantine tolerance was unnecessary (trusted infra), so I avoided its 3f+1 overhead.”

Template: state failure model + RTO/RPO → redundancy across independent domains → idempotency for safety → per-dependency resilience (timeout/retry/breaker/bulkhead) → containment (cells) → MTTR + chaos validation → state the tradeoff.

F. Consolidated gotchas & traps (rapid fire)

• Retries amplify outages — budget + backoff + jitter; idempotent only.
• Perfect failure detection is impossible (FLP) — you suspect, not know.
• Cut MTTR — often cheaper than raising MTBF.
• Serial dependencies multiply availability down.
• Parallel redundancy only helps if failures are independent.
• Metastable failure persists after the trigger — break the loop.
• Too-long timeouts → thread-pool exhaustion → cascade.
• Shallow health checks miss dependency failures.
• Byzantine needs 3f+1.
• State RTO and RPO, not just “it fails over.”

G. Pros/cons master tables

Resilience patterns

Redundancy topologies

Go deeper (primary sources)

• Avizienis, Laprie, Randell, Landwehr, “Basic Concepts and Taxonomy of Dependable and Secure Computing” (2004).
• Chandra & Toueg, “Unreliable Failure Detectors for Reliable Distributed Systems” (1996); Hayashibara et al., “The φ Accrual Failure Detector” (2004).
• Lamport, Shostak, Pease, “The Byzantine Generals Problem” (1982).
• Fischer, Lynch, Paterson, “Impossibility of Distributed Consensus with One Faulty Process” (1985).
• Gray, “Why Do Computers Stop and What Can Be Done About It?” (1986).
• Candea & Fox, “Crash-Only Software” (2003).
• Nygard, Release It! (2007) — circuit breakers, bulkheads.
• Basiri et al., “Chaos Engineering” (2016).
• Bronson, Aghayev, Charapko, Zhu, “Metastable Failures in Distributed Systems” (2021).
• Beyer, Jones, Petoff, Murphy, Site Reliability Engineering (2016) — SLIs/SLOs/error budgets.

PART 5 — MAKE IT STICK (Teaching Tutorial)

The references are the map; this is the driving lesson. Reliability becomes intuitive once you can do the availability arithmetic in your head and picture how failures spread. Two formulas and two pictures.

14.1 The one idea: bound probability AND blast radius

   You can't PREVENT failure. You can only:
     (1) make it RARE        → redundancy (lower probability)
     (2) make it SMALL       → isolation + degradation (lower blast radius)

Every technique below is one or the other. When you design for reliability, ask: am I making failure rarer, or smaller? (You usually need both.)

14.2 The availability arithmetic (do this in your head)

   A = MTBF / (MTBF + MTTR)        ← two levers: raise MTBF, or LOWER MTTR (often cheaper)

   SERIAL dependencies (need all):     A_total = A1 × A2 × A3 …   ← each one DRAGS YOU DOWN
        5 services @ 99.9%  →  0.999^5 ≈ 99.5%   (you got WORSE)

   PARALLEL redundancy (any survives): unavail = (1-A1) × (1-A2) …  ← multiplies FAILURE away
        two @ 99%  →  0.01 × 0.01 = 0.0001 →  99.99%   (huge win — IF independent)

Two memory hooks: every dependency you add multiplies your availability down; every independent replica multiplies your failure probability away. And lowering MTTR (faster recovery) is usually a cheaper nine than raising MTBF. The catch: parallel math assumes independent failures — a bad deploy or shared AZ breaks that.

14.3 Cascading failure (how one slow service kills everything)

  Dependency D slows to 5 s →
     callers' threads BLOCK waiting on D →
        thread pool EXHAUSTED →
           caller can't serve UNRELATED requests → it "fails" →
              ITS callers' pools exhaust →  … outage propagates UP the graph
  Cut it with: short TIMEOUTS + CIRCUIT BREAKER (stop calling D) + BULKHEAD (cap D's threads)

14.4 Metastable failure (the scary one: trigger gone, still down)

  brief overload → clients RETRY → retries BECOME the load → still overloaded →
     more retries → … STABLE BAD STATE that persists EVEN AFTER the trigger is gone.
  Removing the original cause does NOT fix it. You must BREAK THE LOOP:
     shed load, disable/limit retries, drain queues, THEN ramp back up.

If you ever hear “the cause is fixed but it’s still down,” think metastable and break the feedback loop. Prevent it with retry budgets + backoff + jitter and load shedding.

14.5 The nines, felt (so they mean something)

  99%     → 3.65 days/yr down       99.99%  → 52.6 min/yr
  99.9%   → 8.76 hours/yr           99.999% → 5.26 min/yr   (chase only if it pays)

14.6 Analogies that stick

• Bulkhead = ship compartments — flood one, the ship floats. (Separate thread pools per dependency.)
• Circuit breaker = the one in your house — trips to stop a fault from burning the place down, then you reset it.
• Retry storm = everyone redialing a busy line at once — the redials are the busy signal.
• Cells = lifeboats — a failure sinks one boat’s passengers, not the whole ship.

14.7 Misconceptions → corrections

14.8 Explain it back (Feynman)

1. Every technique makes failure rare or small — classify three of them. [14.1]
2. Why do 5 serial 99.9% deps give <99.9%? Why do 2 parallel 99% give 99.99%? [14.2]
3. Trace a cascading failure and three cuts. [14.3]
4. What’s metastable failure and how do you recover? [14.4]
5. Crash vs Byzantine node counts? [cheatsheet]

14.9 Flashcards (cover the right column)

14.10 The 60-second recall

“You can’t prevent failure, only make it rare (redundancy) and small (isolation, degradation). Availability is MTBF over MTBF-plus-MTTR, and lowering recovery time is usually a cheaper nine than raising reliability. Serial dependencies multiply availability down — every dependency hurts — while independent parallel replicas multiply failure probability away, but only if failures are truly independent. A slow dependency causes cascading failure by exhausting thread pools, so cut it with short timeouts, circuit breakers, and bulkheads. Beware metastable failure, where retries become the load and the system stays down after the trigger is gone — recover by breaking the loop (shed load, kill retries, drain), and prevent it with retry budgets, backoff, and jitter. Always state RTO and RPO, not just ‘it fails over.’”

Frequently asked questions

What’s the difference between fault, error, and failure?

A fault is the root defect (a bug, a bad disk), an error is the resulting incorrect internal state, and a failure is user-visible deviation from specification. Resilience breaks this chain by tolerating errors so they never become failures — a fault that is masked never becomes a failure.

Why do retries amplify outages?

During an incident, retries add load exactly when the system is already overloaded, pushing utilization toward saturation and turning a brownout into an outage (a retry storm). Always bound retries with a budget, use exponential backoff with jitter, and require idempotency so retries are safe.

Is it better to increase MTBF or reduce MTTR?

Availability equals MTBF divided by (MTBF plus MTTR), so both help, but reducing mean time to recovery — through fast detection, automated failover, and quick rollback — is usually the cheaper path to additional nines than making components more reliable.

What is a metastable failure?

A metastable failure is a stable bad state that persists even after the original trigger is removed, because a feedback loop (often retries that became the load) sustains it. Recovery requires breaking the loop — shedding load, disabling retries, draining queues — not just removing the original cause.

How many nodes tolerate f crash versus Byzantine faults?

To tolerate f crash (fail-stop) faults you need 2f+1 nodes with majority quorums. To tolerate f Byzantine (arbitrary or malicious) faults you need 3f+1 nodes with quorums of 2f+1, because a lying node can send conflicting information to different peers.