Fault Tolerance

Fault tolerance: designing for bad days

Imagine a food delivery app at 7:30 PM: one database crashes, a network link drops, and a machine reboots mid-order. If the whole service goes down, the system is fragile. If users can still browse restaurants and place orders, the system is fault tolerant.

In distributed systems, failures are normal, not rare. Machines fail, messages arrive late, clocks disagree, and entire zones can disappear. Fault tolerance means the system continues to provide acceptable service even when some components fail.

This does not mean “nothing ever breaks.” It means:

• failures are expected
• the system can detect them
• work can shift elsewhere
• data stays consistent enough for the goal

Think of it like a plane with multiple engines: not failure-free, but built to survive failure.

The main tools: redundancy, detection, and recovery

Fault tolerance usually comes from redundancy: having backups for computation, storage, or network paths. If one replica fails, another takes over. A common pattern is 3 replicas of a service across different machines or zones.

But copies alone are not enough. The system must also:

• detect failure using heartbeats or timeouts
• redirect traffic with a load balancer or leader election
• recover state from logs, replicas, or checkpoints

For example, if a leader node stops sending heartbeats for 5 seconds, followers may elect a new leader. That avoids a single point of failure.

A simple availability estimate shows why redundancy helps:

availability = uptime / total time

If one server is available 99% of the time, two independent replicas behind failover are unavailable only when both are down:

0.01 x 0.01 = 0.0001

So combined availability is about 99.99%.

Common failure-handling techniques in practice

Distributed systems use several practical patterns to survive faults. Retries handle temporary network glitches. Timeouts prevent waiting forever. Replication preserves service and data. Circuit breakers stop a failing dependency from dragging down everything else.

Here is a tiny example of retry logic:

for attempt in range(3):
    try:
        return call_inventory_service(timeout=0.2)
    except TimeoutError:
        sleep(0.05 * (attempt + 1))
return "inventory temporarily unavailable"

This works well for short-lived failures, but retries can also amplify overload if every client retries at once. That is why systems often add backoff and jitter.

Another key idea is graceful degradation: if recommendations fail, an e-commerce site can still let users search and buy. Better partial service than total outage.

Fault tolerance always involves trade-offs

Here is the catch: handling failures often costs money, latency, and complexity. Replicating data to 3 zones is safer than storing it once, but writes may become slower. Automatic failover is powerful, but if two nodes both think they are leader, you can corrupt data.

So engineers choose what must survive:

• a single machine failure?
• a rack failure?
• an entire availability zone outage?
• temporary stale data, or never?

A chat app may accept a few seconds of stale presence status. A banking ledger usually cannot.

The core mindset is simple: don’t ask, “How do we prevent all failures?” Ask, “When failure happens, how do we keep the important parts working safely?” That question is the heart of fault tolerance.