Skip to content
System Design Lab
Design to Scale

Autoscaling

Add instances when load rises and remove them when it falls — and learn why warm-up time, step size, cooldowns and the choice of metric decide whether autoscaling saves you or flaps.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Explain the autoscaling loop: metric, threshold, action, warm-up, cooldown.
  • See why slow, one-at-a-time scaling loses a sudden spike.
  • Recognise flapping and prevent it with hysteresis and cooldowns.
  • Choose between CPU, request-rate, queue-depth and scheduled scaling.

Before you start

Overview

A supermarket opens more checkouts when queues get long and closes them when it is quiet. It works if staff can get to a till quickly — and if the manager does not open and close the same till every two minutes.

An autoscaler runs a loop: measure a metric (CPU, requests per instance, queue depth), compare it with thresholds, act by adding or removing instances, then wait. New instances need a warm-up (boot, JIT, cache fill) before they help; a cooldown stops the loop from reacting to its own previous change.

Run it

Traffic jumps from 100 to 500 requests/s at t = 10–13 s and falls back at t = 40 s; one instance handles ~80/s. Watch the Instances chart against the failures in each policy, then drag the warm-up time.

Loading simulation…

What to scale on

MetricGood forWatch out
CPU utilisationCPU-bound web servicesLags for I/O-bound services; saturates at 100% and stops telling you how far over you are.
Requests per instanceStateless HTTP servicesNeeds a known per-instance capacity.
Queue depth / lagWorkers and consumersScale-in must not kill work in progress; capped by partitions for Kafka.
Schedule / predictionKnown daily peaks, eventsAdd capacity before the spike instead of during it.

Kubernetes HPA

Scale fast, shrink slowly
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies: [{ type: Percent, value: 100, periodSeconds: 15 }]   # double quickly
    scaleDown:
      stabilizationWindowSeconds: 300                                # wait 5 min before shrinking
      policies: [{ type: Pods, value: 1, periodSeconds: 60 }]

Trade-offs

BenefitsCosts
CostPay for peak capacity only during the peak.Low targets and high minimums trade money for safety.
ResponsivenessLarge steps and short cooldowns react fast.Too aggressive settings overshoot and flap.
StabilityHysteresis and slow scale-in keep fleets steady.Slow policies lose sudden spikes.

Production considerations

  • Make instances stateless and readiness-checked so new ones only get traffic when warm.
  • Keep a gap between scale-out and scale-in thresholds (hysteresis), and scale in slowly.
  • Check downstream limits: doubling web instances also doubles database connections.
  • Load-test the scaling policy itself, not just the application.

Further reading

Related topics