Skip to content
System Design Lab
Design to Scale

Load Balancing

Spread requests across instances so none is overloaded — and see how the choice of algorithm and health checks decides your tail latency when instances are not identical.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Compare round robin, random and least connections on a mixed fleet.
  • Explain why tail latency (p99) reveals a bad balancing decision before averages do.
  • Use passive and active health checks to stop sending traffic to dead instances.
  • Know the difference between layer-4 and layer-7 balancing.

Before you start

Requirements

Three instances serve 80 requests per second. Two are new (60 ms per request, 4 workers each); the third runs on older hardware and needs 150 ms. Fleets are rarely uniform: noisy neighbours, a JVM still warming up, a node doing garbage collection or a slower availability zone all create the same situation.

Think of supermarket checkouts. Sending shoppers to lanes in strict rotation ignores that one cashier is slow, so that lane's queue keeps growing. Sending each shopper to the shortest queue fixes it without anyone measuring the cashiers' speed.

The algorithms

AlgorithmHow it picksStrengthWeakness
Round robinNext instance in a fixed rotation.Simple, even spread for identical instances.Blind to speed and current load.
RandomUniformly random instance.Stateless; works with many balancers.Same blindness, plus short-term imbalance.
Least connectionsInstance with the fewest requests in flight.Adapts to slow instances automatically.Needs per-instance counters; fast-failing instances look attractive.
Power of two choicesPick two at random, send to the less loaded.Near least-connections quality with little coordination.Slightly less optimal than full least connections.
Consistent hashingHash of a key (user, session) picks the instance.Cache affinity; stable placement.Hot keys overload one instance.

Run it: same fleet, different algorithms

Compare the p99 tile across scenarios A, B and C. Averages look similar; the tail tells the truth. The in-flight chart shows round robin piling requests onto app-3 while least connections keeps everyone level.

Loading simulation…

Health checks

  • Active — the balancer probes /health every few seconds and removes instances that fail several probes. Catches dead instances even with no traffic.
  • Passive (outlier detection) — the balancer watches real responses and ejects an instance after consecutive errors or timeouts. Reacts within requests, not seconds.
  • Use both. Make the health endpoint check what the instance itself needs to serve (not every downstream dependency — otherwise one database blip ejects the whole fleet).

Layer 4 vs layer 7 Advanced

Advanced A layer-4 balancer forwards TCP connections without reading them: very fast, but with HTTP/2 or gRPC many requests share one long-lived connection, so per-connection balancing can pin a busy client to one instance. A layer-7 balancer terminates HTTP and balances each request, can route by path or header, retry idempotent requests and run outlier detection — at the cost of more CPU and TLS termination at the balancer.

Configuration example

nginx.conf
upstream orders_api {
    least_conn;                                   # least connections instead of round robin
    server 10.0.1.11:8080 max_fails=3 fail_timeout=5s;   # passive health check
    server 10.0.1.12:8080 max_fails=3 fail_timeout=5s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=5s;
    keepalive 64;
}

server {
    listen 443 ssl;
    location /orders/ {
        proxy_pass http://orders_api;
        proxy_next_upstream error timeout http_502 http_503;   # retry another instance
        proxy_next_upstream_tries 2;
        proxy_read_timeout 2s;
    }
}

Trade-offs

BenefitsCosts
Round robinPredictable and trivial.Tail latency suffers as soon as instances differ.
Least connectionsSelf-correcting under uneven load.Attracts traffic to fast-failing instances unless health checks eject them.
Layer 7Per-request decisions, routing, retries, outlier detection.More CPU and an extra TLS termination point.

Production considerations

  • Run the balancer itself redundantly (managed cloud balancers do this for you).
  • Drain connections during deployments so in-flight requests finish before an instance stops.
  • Watch per-instance latency and error rate, not only the fleet average — that is where a bad instance hides.

Further reading

Related topics