Design to Scale
Capacity Planning
Estimate how much capacity you need before users find out for you: utilisation, queueing and Little's law explain why latency explodes near 100% and how much headroom to keep.
What you will learn
- Calculate capacity from service time and concurrency.
- Explain why latency grows sharply as utilisation approaches 100%.
- Use Little's law to relate traffic, latency and concurrency.
- Plan headroom for peaks, failures and growth.
Before you start
Overview
A motorway at 50% of capacity flows freely; at 90% a single braking car causes a jam. Servers behave the same way: requests arrive in random clumps, and the fuller the system, the longer the clumps have to wait.
Three simple relationships do most of the work. Capacity = workers ÷ service time (8 workers × 50 ms ⇒ 160 req/s). Utilisation = traffic ÷ capacity. Little's law: requests in flight = throughput × latency — so 100 req/s at 200 ms means 20 requests in progress at any moment, which sizes your thread pools and connection pools.
Try the numbers
Capacity calculator
Queueing maths (M/M/c): random arrivals, random service times, identical workers.
- Capacity at 100%
- 160 req/s
- Utilisation
- 75%
- Requests that must queue
- 36%
- Average response time
- 59 ms
- p95 response time (approx.)
- 199 ms
- In flight (Little's law)
- 7.1
You need 9 workers to stay at or below 70% utilisation.
Average response time as traffic approaches capacity; the dot is your setting, the dashed line 70%.
Run it
The same service under steadily rising traffic. Watch p95 in A as utilisation passes 70%, 90% and 100%, then compare B with 50% more workers.
How much headroom
- Latency: keep normal utilisation around 50–70% for latency-sensitive services; beyond ~80% queueing dominates.
- Peaks: plan for the peak hour (or minute), not the daily average — often 2–5× higher.
- Failures (N+1): with N instances needed, run N+1 (or survive losing a zone: N × zones / (zones − 1)).
- Growth: add the expected growth until the next time you can add capacity.
A practical process
- Measure: peak requests per second per endpoint and their latency at low load (the service time).
- Load-test one instance to find where latency starts to bend — that is its practical capacity, not the point where it breaks.
- Compute instances = peak ÷ (capacity per instance × target utilisation), then add N+1 and growth.
- Repeat for every dependency: database connections, downstream rate limits, broker partitions.
- Review after every launch and every incident.
k6 run --vus 1 --stage 2m:20,2m:40,2m:60,2m:80,2m:100 checkout.js
# Plot p95 against requests/s: capacity is where p95 starts to rise steeply, not where errors begin.Trade-offs
| Benefits | Costs | |
|---|---|---|
| Headroom | Stable latency, room for spikes and failures. | Idle capacity costs money. |
| Running hot | Cheaper per request. | Small spikes cause large latency jumps and timeouts. |
| Models | Queueing maths gives quick, explainable estimates. | Real systems have contention and skew — always confirm with load tests. |
Production considerations
- Track utilisation and saturation (queue length) per resource, alongside latency.
- Remember pools multiply: 20 instances × 30 connections = 600 database connections.
- Combine planning with autoscaling — plan the minimum and maximum, let the autoscaler move in between.
- Make overload safe with backpressure so mistakes in the plan degrade gracefully.