Skip to content
System Design Lab
Design to Scale

Vertical vs Horizontal Scaling

Grow a web application from one server to a fleet: what breaks at each step, why scaling up hits a ceiling, why scaling out needs stateless instances, and where the next bottleneck appears.

Beginner 6 min read· Updated 2026-11-18

What you will learn

  • Estimate a server's capacity from its workers and request latency.
  • Compare scaling up (bigger machine) with scaling out (more machines).
  • Explain why horizontal scaling requires stateless application instances.
  • Recognise that removing one bottleneck exposes the next one.

Requirements

A product API serves 30 requests per second today. A launch is coming and traffic will reach 120 requests per second. Each request takes about 80 ms of server time. Users give up after 3 seconds. We want every request to succeed with a p95 under 300 ms.

Scaling evolution

Each stage below runs the same traffic ramp — 30, 60, 90 and finally 120 requests per second — against a different architecture. Step through them in order and press Play on each.

Stage 1 of 4

One server

Current architecture
Users call a single application server with 4 worker threads.
Limitation
Capacity is fixed at ~50 requests/s and the server is a single point of failure.
Bottleneck
At 60 requests/s the queue fills; latency jumps and requests are rejected. CPU-bound workers cannot go faster.
Improvement
Give the server more resources (scale up) or add more servers (scale out).
Performance impact
Fine until ~50 requests/s, then latency grows without bound.
Complexity & operational cost
Cheapest and simplest possible setup.
Trade-off
Simplicity now, an outage at the first traffic spike.
Loading simulation…

Why scale-out needs stateless instances

If instance 1 keeps a user's session in memory and the next request lands on instance 2, the user is logged out. Sticky sessions hide the problem but break load distribution and lose sessions when an instance dies. Keep instances disposable: any instance can serve any request, and killing one loses nothing but in-flight work.

  • Sessions → a shared store (Redis) or stateless tokens (JWT).
  • Uploaded files → object storage (S3, GCS), not the local disk.
  • Scheduled jobs → a single leader or a distributed lock, so 3 instances do not send 3 emails.
  • In-memory caches → fine for derived data, never for the source of truth.

Autoscaling and capacity planning Advanced

Advanced Autoscalers add instances when a metric (CPU, concurrency, queue length) crosses a target and remove them when it falls. New instances take time to start and warm up, so scale on a target below saturation (e.g. 60–70% CPU), keep a minimum of at least two instances for redundancy, and size the database and downstream dependencies for the maximum instance count — or the autoscaler will happily overload them.

Trade-offs

BenefitsCosts
Scale upNo architectural change; one machine to operate.Hard ceiling, non-linear cost, single point of failure.
Scale outNear-linear capacity and built-in redundancy.Requires stateless design, a load balancer and fleet operations.
BothBuys time to fix the real bottleneck.Neither helps if the shared database is the limit.

Final architecture

UsersUsersLoad Balancerround robinLoad Balancerapp-14 workersapp-1app-24 workersapp-2app-34 workersapp-3PostgreSQLpool: 6 connectio…PostgreSQL
Stateless instances behind a load balancer — with the database as the component to scale next.

Further reading

Related topics