Design to Scale
High Availability
Remove single points of failure with redundant components and automatic failover — and tune failure detection so it reacts to real crashes, not to every network hiccup.
What you will learn
- Express availability as nines and allowed downtime.
- Explain active-passive failover: detection, promotion, RTO and RPO.
- Balance detection speed against false failovers.
- Find single points of failure in an architecture.
Before you start
Overview
A hospital keeps a generator that starts automatically when the mains power fails. Nobody needs to notice the outage and run to the basement — but the generator must not start every time a light flickers.
High availability means the service keeps working when a component fails. Every component on the request path is duplicated, and something detects failures and moves traffic: a load balancer, a virtual IP, a database proxy or a cluster manager. Two numbers describe the result: RTO (how long recovery takes) and RPO (how much recent data can be lost).
| Availability | Downtime per year | Downtime per month |
|---|---|---|
| 99% (two nines) | 3.65 days | 7.3 hours |
| 99.9% | 8.8 hours | 43.8 minutes |
| 99.99% | 52.6 minutes | 4.4 minutes |
| 99.999% | 5.3 minutes | 26 seconds |
Run it
Two short network blips (t = 6 s, 13 s) and a real crash (t = 22 s). Compare Seconds with errors and Failovers, and follow the serving node on the timeline.
Detecting failure
- Detection time ≈ probe interval × failure threshold. 1 s × 3 = ~3 s before anything happens.
- Too slow and every real failure costs more downtime; too fast and transient blips (GC pauses, network jitter) trigger failovers that cause their own outage.
- Failback should usually be manual or delayed: moving back too eagerly doubles the disruption.
- Split brain: if the old primary is alive but unreachable, two nodes may accept writes. Use fencing (STONITH, leases, quorum) so only one primary can exist.
Example: database failover behind a proxy
bootstrap:
dcs:
ttl: 30 # leader lease: a primary that cannot renew it demotes itself (fencing)
loop_wait: 10 # health loop interval
retry_timeout: 10
maximum_lag_on_failover: 1048576 # don't promote a replica more than 1 MB behind (bounds RPO)
synchronous_mode: false # true = RPO 0, at the cost of write latency and availabilityTrade-offs
| Benefits | Costs | |
|---|---|---|
| Availability | Single failures cost seconds instead of hours. | Failover itself is a risky operation that must be tested. |
| Cost | Standbys can serve reads or batch work. | At least double the infrastructure for the redundant parts. |
| Data | Synchronous replication gives RPO = 0. | It adds write latency and can block writes if the standby is unavailable. |
Production considerations
- Draw the request path and mark every single point of failure, including DNS, load balancers and secrets stores.
- Run failover drills regularly (game days); an untested failover usually fails.
- Alert on the standby's health and lag, not only the primary's.
- Set availability targets per service from user needs and error budgets, not "as high as possible".