Skip to content
System Design Lab
Microservice Patterns

Queue-Based Load Leveling

Put a queue between a bursty producer and a system that cannot scale, so the system works at a steady pace it can sustain while the queue absorbs the peaks.

Beginner 4 min read· Updated 2026-11-18

What you will learn

  • Protect a fixed-capacity system (legacy API, partner, database) from traffic spikes.
  • Cap downstream concurrency with the number of workers.
  • Check whether a queue can drain: average arrival rate vs capacity.
  • Decide which operations may become asynchronous.

Overview

A dam turns a river's flash floods into a steady flow that the turbines and fields downstream can handle. The reservoir fills during the storm and drains slowly afterwards. A queue is a reservoir for requests.

Some systems cannot scale: a legacy ERP with four connections, a partner API with a strict quota, a database sized for normal days. Queue-based load leveling decouples the producer's rate from the consumer's rate. Producers enqueue work quickly; a fixed number of workers feed the fragile system at a rate it can take.

Run it

Compare the Legacy API utilization chart between A and B. In A it pins at 100% and rejects writes during the spike. In B it stays flat and below 100% while the queue fills and drains. Scenario C is the limit of the pattern.

Loading simulation…

When does it work?

  • Average below capacity: peaks above capacity are fine; a sustained average above it only grows the backlog (scenario C). Then you need more capacity or rate limiting at the edge.
  • The caller can wait: the user gets "accepted" now and the result later (email, notification, status page).
  • Work is retryable and idempotent: queued work may be redelivered; see Idempotent Consumer.

Implementation

ErpWriter.java
@Component
class ErpWriter {

    private final ErpClient erp;

    ErpWriter(ErpClient erp) {
        this.erp = erp;
    }

    // Exactly 3 concurrent writes to the ERP across this instance — the ERP's safe limit.
    // Run a single instance (or divide the limit across instances).
    @RabbitListener(queues = "erp-writes", concurrency = "3")
    public void write(ErpOrder order) {
        erp.upsertOrder(order);   // idempotent upsert keyed by order id
    }
}

Advanced With autoscaled workers the concurrency limit multiplies by instance count, which defeats the purpose. Either pin the worker count, or enforce the downstream limit with a distributed semaphore or rate limiter shared by all workers.

Trade-offs

BenefitsCosts
ProtectionFragile systems see steady, bounded load.Work is delayed during peaks — by the backlog divided by the drain rate.
AvailabilityProducers keep working when the downstream is slow or briefly down.A queue outage now blocks producers; the broker must be highly available.
DesignSimple to add in front of existing systems.Turns synchronous answers into asynchronous ones.

Production considerations

  • Alert on the age of the oldest message, and estimate drain time = backlog ÷ (capacity − arrival rate).
  • Bound the queue and reject or shed load when it is full, rather than accepting work you cannot finish in time.
  • Separate queues by priority so a bulk import cannot delay customer orders.

Further reading

Related topics