Microservice Patterns
Queue-Based Load Leveling
Put a queue between a bursty producer and a system that cannot scale, so the system works at a steady pace it can sustain while the queue absorbs the peaks.
What you will learn
- Protect a fixed-capacity system (legacy API, partner, database) from traffic spikes.
- Cap downstream concurrency with the number of workers.
- Check whether a queue can drain: average arrival rate vs capacity.
- Decide which operations may become asynchronous.
Overview
A dam turns a river's flash floods into a steady flow that the turbines and fields downstream can handle. The reservoir fills during the storm and drains slowly afterwards. A queue is a reservoir for requests.
Some systems cannot scale: a legacy ERP with four connections, a partner API with a strict quota, a database sized for normal days. Queue-based load leveling decouples the producer's rate from the consumer's rate. Producers enqueue work quickly; a fixed number of workers feed the fragile system at a rate it can take.
Run it
Compare the Legacy API utilization chart between A and B. In A it pins at 100% and rejects writes during the spike. In B it stays flat and below 100% while the queue fills and drains. Scenario C is the limit of the pattern.
When does it work?
- Average below capacity: peaks above capacity are fine; a sustained average above it only grows the backlog (scenario C). Then you need more capacity or rate limiting at the edge.
- The caller can wait: the user gets "accepted" now and the result later (email, notification, status page).
- Work is retryable and idempotent: queued work may be redelivered; see Idempotent Consumer.
Implementation
@Component
class ErpWriter {
private final ErpClient erp;
ErpWriter(ErpClient erp) {
this.erp = erp;
}
// Exactly 3 concurrent writes to the ERP across this instance — the ERP's safe limit.
// Run a single instance (or divide the limit across instances).
@RabbitListener(queues = "erp-writes", concurrency = "3")
public void write(ErpOrder order) {
erp.upsertOrder(order); // idempotent upsert keyed by order id
}
}Advanced With autoscaled workers the concurrency limit multiplies by instance count, which defeats the purpose. Either pin the worker count, or enforce the downstream limit with a distributed semaphore or rate limiter shared by all workers.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Protection | Fragile systems see steady, bounded load. | Work is delayed during peaks — by the backlog divided by the drain rate. |
| Availability | Producers keep working when the downstream is slow or briefly down. | A queue outage now blocks producers; the broker must be highly available. |
| Design | Simple to add in front of existing systems. | Turns synchronous answers into asynchronous ones. |
Production considerations
- Alert on the age of the oldest message, and estimate drain time = backlog ÷ (capacity − arrival rate).
- Bound the queue and reject or shed load when it is full, rather than accepting work you cannot finish in time.
- Separate queues by priority so a bulk import cannot delay customer orders.