Design to Scale
Autoscaling
Add instances when load rises and remove them when it falls — and learn why warm-up time, step size, cooldowns and the choice of metric decide whether autoscaling saves you or flaps.
What you will learn
- Explain the autoscaling loop: metric, threshold, action, warm-up, cooldown.
- See why slow, one-at-a-time scaling loses a sudden spike.
- Recognise flapping and prevent it with hysteresis and cooldowns.
- Choose between CPU, request-rate, queue-depth and scheduled scaling.
Before you start
Overview
A supermarket opens more checkouts when queues get long and closes them when it is quiet. It works if staff can get to a till quickly — and if the manager does not open and close the same till every two minutes.
An autoscaler runs a loop: measure a metric (CPU, requests per instance, queue depth), compare it with thresholds, act by adding or removing instances, then wait. New instances need a warm-up (boot, JIT, cache fill) before they help; a cooldown stops the loop from reacting to its own previous change.
Run it
Traffic jumps from 100 to 500 requests/s at t = 10–13 s and falls back at t = 40 s; one instance handles ~80/s. Watch the Instances chart against the failures in each policy, then drag the warm-up time.
What to scale on
| Metric | Good for | Watch out |
|---|---|---|
| CPU utilisation | CPU-bound web services | Lags for I/O-bound services; saturates at 100% and stops telling you how far over you are. |
| Requests per instance | Stateless HTTP services | Needs a known per-instance capacity. |
| Queue depth / lag | Workers and consumers | Scale-in must not kill work in progress; capped by partitions for Kafka. |
| Schedule / prediction | Known daily peaks, events | Add capacity before the spike instead of during it. |
Kubernetes HPA
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
minReplicas: 3
maxReplicas: 30
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies: [{ type: Percent, value: 100, periodSeconds: 15 }] # double quickly
scaleDown:
stabilizationWindowSeconds: 300 # wait 5 min before shrinking
policies: [{ type: Pods, value: 1, periodSeconds: 60 }]Trade-offs
| Benefits | Costs | |
|---|---|---|
| Cost | Pay for peak capacity only during the peak. | Low targets and high minimums trade money for safety. |
| Responsiveness | Large steps and short cooldowns react fast. | Too aggressive settings overshoot and flap. |
| Stability | Hysteresis and slow scale-in keep fleets steady. | Slow policies lose sudden spikes. |
Production considerations
- Make instances stateless and readiness-checked so new ones only get traffic when warm.
- Keep a gap between scale-out and scale-in thresholds (hysteresis), and scale in slowly.
- Check downstream limits: doubling web instances also doubles database connections.
- Load-test the scaling policy itself, not just the application.