Design to Scale
Load Balancing
Spread requests across instances so none is overloaded — and see how the choice of algorithm and health checks decides your tail latency when instances are not identical.
What you will learn
- Compare round robin, random and least connections on a mixed fleet.
- Explain why tail latency (p99) reveals a bad balancing decision before averages do.
- Use passive and active health checks to stop sending traffic to dead instances.
- Know the difference between layer-4 and layer-7 balancing.
Before you start
Requirements
Three instances serve 80 requests per second. Two are new (60 ms per request, 4 workers each); the third runs on older hardware and needs 150 ms. Fleets are rarely uniform: noisy neighbours, a JVM still warming up, a node doing garbage collection or a slower availability zone all create the same situation.
Think of supermarket checkouts. Sending shoppers to lanes in strict rotation ignores that one cashier is slow, so that lane's queue keeps growing. Sending each shopper to the shortest queue fixes it without anyone measuring the cashiers' speed.
The algorithms
| Algorithm | How it picks | Strength | Weakness |
|---|---|---|---|
| Round robin | Next instance in a fixed rotation. | Simple, even spread for identical instances. | Blind to speed and current load. |
| Random | Uniformly random instance. | Stateless; works with many balancers. | Same blindness, plus short-term imbalance. |
| Least connections | Instance with the fewest requests in flight. | Adapts to slow instances automatically. | Needs per-instance counters; fast-failing instances look attractive. |
| Power of two choices | Pick two at random, send to the less loaded. | Near least-connections quality with little coordination. | Slightly less optimal than full least connections. |
| Consistent hashing | Hash of a key (user, session) picks the instance. | Cache affinity; stable placement. | Hot keys overload one instance. |
Run it: same fleet, different algorithms
Compare the p99 tile across scenarios A, B and C. Averages look similar; the tail tells the truth. The in-flight chart shows round robin piling requests onto app-3 while least connections keeps everyone level.
Health checks
- Active — the balancer probes
/healthevery few seconds and removes instances that fail several probes. Catches dead instances even with no traffic. - Passive (outlier detection) — the balancer watches real responses and ejects an instance after consecutive errors or timeouts. Reacts within requests, not seconds.
- Use both. Make the health endpoint check what the instance itself needs to serve (not every downstream dependency — otherwise one database blip ejects the whole fleet).
Layer 4 vs layer 7 Advanced
Advanced A layer-4 balancer forwards TCP connections without reading them: very fast, but with HTTP/2 or gRPC many requests share one long-lived connection, so per-connection balancing can pin a busy client to one instance. A layer-7 balancer terminates HTTP and balances each request, can route by path or header, retry idempotent requests and run outlier detection — at the cost of more CPU and TLS termination at the balancer.
Configuration example
upstream orders_api {
least_conn; # least connections instead of round robin
server 10.0.1.11:8080 max_fails=3 fail_timeout=5s; # passive health check
server 10.0.1.12:8080 max_fails=3 fail_timeout=5s;
server 10.0.1.13:8080 max_fails=3 fail_timeout=5s;
keepalive 64;
}
server {
listen 443 ssl;
location /orders/ {
proxy_pass http://orders_api;
proxy_next_upstream error timeout http_502 http_503; # retry another instance
proxy_next_upstream_tries 2;
proxy_read_timeout 2s;
}
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Round robin | Predictable and trivial. | Tail latency suffers as soon as instances differ. |
| Least connections | Self-correcting under uneven load. | Attracts traffic to fast-failing instances unless health checks eject them. |
| Layer 7 | Per-request decisions, routing, retries, outlier detection. | More CPU and an extra TLS termination point. |
Production considerations
- Run the balancer itself redundantly (managed cloud balancers do this for you).
- Drain connections during deployments so in-flight requests finish before an instance stops.
- Watch per-instance latency and error rate, not only the fleet average — that is where a bad instance hides.