Microservice Patterns
Rate Limiting
Cap how many requests a client may make per unit of time to protect services from bursts and abuse — and pick the algorithm whose behaviour at the edges you can live with.
What you will learn
- Compare fixed window, sliding window, token bucket and leaky bucket on the same traffic.
- Explain the fixed-window boundary burst and why it overloads backends.
- Choose between rejecting (429) and shaping (queueing) excess traffic.
- Implement distributed rate limiting with Redis and Spring.
Overview
A nightclub bouncer lets in a certain number of people per minute. Some bouncers count per clock minute, some keep a rolling count, some hand out tickets that refill over time, and some make everyone queue and admit one person every few seconds. All of them limit the rate — but they behave very differently when a crowd arrives at once.
A rate limiter counts requests per client (API key, user, IP) and refuses — or delays — requests above an agreed rate. It protects services from overload, keeps one noisy client from starving others, and enforces quotas of paid plans.
Four algorithms
| Algorithm | Rule | At a burst |
|---|---|---|
| Fixed window | At most N requests per calendar interval; the counter resets at the boundary. | Up to 2N can pass around a boundary (end of one window + start of the next). |
| Sliding window | Weighted count of the current and previous window approximates "N in any interval". | Close to N in any rolling interval. Small memory cost. |
| Token bucket | A bucket of B tokens refills at R per second; each request spends one. | Allows a burst of B, then R per second. Most common for public APIs. |
| Leaky bucket | Requests queue and leave at a constant rate R. | Output is perfectly smooth; excess waits (latency) or is dropped when the queue is full. |
Run it: same bursts, four limiters
Every scenario sends bursts of 12 requests just before and just after a second boundary, against a limit of 10 per second. Watch Peak requests in any 1 s and Overloaded backend rejected: the fixed window lets about twice the limit through and the fragile Search API pays for it.
Implementation
@Component
class ApiKeyRateLimitFilter extends OncePerRequestFilter {
private final Map<String, Bucket> buckets = new ConcurrentHashMap<>();
private Bucket newBucket() {
return Bucket.builder()
.addLimit(Bandwidth.builder().capacity(10).refillGreedy(10, Duration.ofSeconds(1)).build())
.build();
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
String apiKey = Optional.ofNullable(request.getHeader("X-Api-Key")).orElse(request.getRemoteAddr());
ConsumptionProbe probe = buckets.computeIfAbsent(apiKey, k -> newBucket()).tryConsumeAndReturnRemaining(1);
if (probe.isConsumed()) {
response.setHeader("X-RateLimit-Remaining", String.valueOf(probe.getRemainingTokens()));
chain.doFilter(request, response);
return;
}
long waitSeconds = Math.max(1, TimeUnit.NANOSECONDS.toSeconds(probe.getNanosToWaitForRefill()));
response.setStatus(429);
response.setHeader("Retry-After", String.valueOf(waitSeconds));
}
}Advanced An in-memory limiter is per instance: with 5 replicas a limit of 10/s becomes 50/s. For a global limit keep the counters in Redis (Bucket4j's Redis integrations, or Spring Cloud Gateway's RequestRateLimiter, which runs a token bucket as an atomic Redis script). Sliding-window counters need two keys per client per window and an atomic increment-and-read.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Fixed window | Trivial: one counter per client. | Boundary bursts of up to 2× the limit. |
| Sliding window | Accurate without storing every timestamp. | An approximation; slightly more state. |
| Token bucket | Explicit burst allowance, easy to explain to API users. | Bursts still reach the backend (up to the bucket size). |
| Leaky bucket | Smooth, predictable load downstream. | Adds latency; a full queue still drops requests. |
Production considerations
- Return
429 Too Many RequestswithRetry-Afterand rate-limit headers so well-behaved clients slow down instead of retrying blindly (see Retry). - Limit per identity (API key, user) rather than per IP where possible; NAT puts many users behind one address.
- Layer limits: a coarse global limit at the edge, finer per-endpoint limits in services.
- Exempt health checks and internal traffic, and alert on clients that are constantly throttled.