Skip to content
System Design Lab
Design to Scale

Capacity Planning

Estimate how much capacity you need before users find out for you: utilisation, queueing and Little's law explain why latency explodes near 100% and how much headroom to keep.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Calculate capacity from service time and concurrency.
  • Explain why latency grows sharply as utilisation approaches 100%.
  • Use Little's law to relate traffic, latency and concurrency.
  • Plan headroom for peaks, failures and growth.

Before you start

Overview

A motorway at 50% of capacity flows freely; at 90% a single braking car causes a jam. Servers behave the same way: requests arrive in random clumps, and the fuller the system, the longer the clumps have to wait.

Three simple relationships do most of the work. Capacity = workers ÷ service time (8 workers × 50 ms ⇒ 160 req/s). Utilisation = traffic ÷ capacity. Little's law: requests in flight = throughput × latency — so 100 req/s at 200 ms means 20 requests in progress at any moment, which sizes your thread pools and connection pools.

Try the numbers

Capacity calculator

Queueing maths (M/M/c): random arrivals, random service times, identical workers.

Capacity at 100%
160 req/s
Utilisation
75%
Requests that must queue
36%
Average response time
59 ms
p95 response time (approx.)
199 ms
In flight (Little's law)
7.1

You need 9 workers to stay at or below 70% utilisation.

70%100%

Average response time as traffic approaches capacity; the dot is your setting, the dashed line 70%.

Run it

The same service under steadily rising traffic. Watch p95 in A as utilisation passes 70%, 90% and 100%, then compare B with 50% more workers.

Loading simulation…

How much headroom

  • Latency: keep normal utilisation around 50–70% for latency-sensitive services; beyond ~80% queueing dominates.
  • Peaks: plan for the peak hour (or minute), not the daily average — often 2–5× higher.
  • Failures (N+1): with N instances needed, run N+1 (or survive losing a zone: N × zones / (zones − 1)).
  • Growth: add the expected growth until the next time you can add capacity.

A practical process

  1. Measure: peak requests per second per endpoint and their latency at low load (the service time).
  2. Load-test one instance to find where latency starts to bend — that is its practical capacity, not the point where it breaks.
  3. Compute instances = peak ÷ (capacity per instance × target utilisation), then add N+1 and growth.
  4. Repeat for every dependency: database connections, downstream rate limits, broker partitions.
  5. Review after every launch and every incident.
Find the knee with a load test (k6)
k6 run --vus 1 --stage 2m:20,2m:40,2m:60,2m:80,2m:100 checkout.js
# Plot p95 against requests/s: capacity is where p95 starts to rise steeply, not where errors begin.

Trade-offs

BenefitsCosts
HeadroomStable latency, room for spikes and failures.Idle capacity costs money.
Running hotCheaper per request.Small spikes cause large latency jumps and timeouts.
ModelsQueueing maths gives quick, explainable estimates.Real systems have contention and skew — always confirm with load tests.

Production considerations

  • Track utilisation and saturation (queue length) per resource, alongside latency.
  • Remember pools multiply: 20 instances × 30 connections = 600 database connections.
  • Combine planning with autoscaling — plan the minimum and maximum, let the autoscaler move in between.
  • Make overload safe with backpressure so mistakes in the plan degrade gracefully.

Further reading

Related topics