Skip to content
System Design Lab
Design to Scale

High Availability

Remove single points of failure with redundant components and automatic failover — and tune failure detection so it reacts to real crashes, not to every network hiccup.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Express availability as nines and allowed downtime.
  • Explain active-passive failover: detection, promotion, RTO and RPO.
  • Balance detection speed against false failovers.
  • Find single points of failure in an architecture.

Before you start

Overview

A hospital keeps a generator that starts automatically when the mains power fails. Nobody needs to notice the outage and run to the basement — but the generator must not start every time a light flickers.

High availability means the service keeps working when a component fails. Every component on the request path is duplicated, and something detects failures and moves traffic: a load balancer, a virtual IP, a database proxy or a cluster manager. Two numbers describe the result: RTO (how long recovery takes) and RPO (how much recent data can be lost).

AvailabilityDowntime per yearDowntime per month
99% (two nines)3.65 days7.3 hours
99.9%8.8 hours43.8 minutes
99.99%52.6 minutes4.4 minutes
99.999%5.3 minutes26 seconds

Run it

Two short network blips (t = 6 s, 13 s) and a real crash (t = 22 s). Compare Seconds with errors and Failovers, and follow the serving node on the timeline.

Loading simulation…

Detecting failure

  • Detection time ≈ probe interval × failure threshold. 1 s × 3 = ~3 s before anything happens.
  • Too slow and every real failure costs more downtime; too fast and transient blips (GC pauses, network jitter) trigger failovers that cause their own outage.
  • Failback should usually be manual or delayed: moving back too eagerly doubles the disruption.
  • Split brain: if the old primary is alive but unreachable, two nodes may accept writes. Use fencing (STONITH, leases, quorum) so only one primary can exist.

Example: database failover behind a proxy

Patroni (PostgreSQL HA) — detection and safety settings
bootstrap:
  dcs:
    ttl: 30                    # leader lease: a primary that cannot renew it demotes itself (fencing)
    loop_wait: 10              # health loop interval
    retry_timeout: 10
    maximum_lag_on_failover: 1048576   # don't promote a replica more than 1 MB behind (bounds RPO)
    synchronous_mode: false            # true = RPO 0, at the cost of write latency and availability

Trade-offs

BenefitsCosts
AvailabilitySingle failures cost seconds instead of hours.Failover itself is a risky operation that must be tested.
CostStandbys can serve reads or batch work.At least double the infrastructure for the redundant parts.
DataSynchronous replication gives RPO = 0.It adds write latency and can block writes if the standby is unavailable.

Production considerations

  • Draw the request path and mark every single point of failure, including DNS, load balancers and secrets stores.
  • Run failover drills regularly (game days); an untested failover usually fails.
  • Alert on the standby's health and lag, not only the primary's.
  • Set availability targets per service from user needs and error budgets, not "as high as possible".

Further reading

Related topics