Skip to content
System Design Lab
Microservice Patterns

Rate Limiting

Cap how many requests a client may make per unit of time to protect services from bursts and abuse — and pick the algorithm whose behaviour at the edges you can live with.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Compare fixed window, sliding window, token bucket and leaky bucket on the same traffic.
  • Explain the fixed-window boundary burst and why it overloads backends.
  • Choose between rejecting (429) and shaping (queueing) excess traffic.
  • Implement distributed rate limiting with Redis and Spring.

Overview

A nightclub bouncer lets in a certain number of people per minute. Some bouncers count per clock minute, some keep a rolling count, some hand out tickets that refill over time, and some make everyone queue and admit one person every few seconds. All of them limit the rate — but they behave very differently when a crowd arrives at once.

A rate limiter counts requests per client (API key, user, IP) and refuses — or delays — requests above an agreed rate. It protects services from overload, keeps one noisy client from starving others, and enforces quotas of paid plans.

Four algorithms

AlgorithmRuleAt a burst
Fixed windowAt most N requests per calendar interval; the counter resets at the boundary.Up to 2N can pass around a boundary (end of one window + start of the next).
Sliding windowWeighted count of the current and previous window approximates "N in any interval".Close to N in any rolling interval. Small memory cost.
Token bucketA bucket of B tokens refills at R per second; each request spends one.Allows a burst of B, then R per second. Most common for public APIs.
Leaky bucketRequests queue and leave at a constant rate R.Output is perfectly smooth; excess waits (latency) or is dropped when the queue is full.

Run it: same bursts, four limiters

Every scenario sends bursts of 12 requests just before and just after a second boundary, against a limit of 10 per second. Watch Peak requests in any 1 s and Overloaded backend rejected: the fixed window lets about twice the limit through and the fragile Search API pays for it.

Loading simulation…

Implementation

Token bucket per API key with Bucket4j
@Component
class ApiKeyRateLimitFilter extends OncePerRequestFilter {

    private final Map<String, Bucket> buckets = new ConcurrentHashMap<>();

    private Bucket newBucket() {
        return Bucket.builder()
                .addLimit(Bandwidth.builder().capacity(10).refillGreedy(10, Duration.ofSeconds(1)).build())
                .build();
    }

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        String apiKey = Optional.ofNullable(request.getHeader("X-Api-Key")).orElse(request.getRemoteAddr());
        ConsumptionProbe probe = buckets.computeIfAbsent(apiKey, k -> newBucket()).tryConsumeAndReturnRemaining(1);
        if (probe.isConsumed()) {
            response.setHeader("X-RateLimit-Remaining", String.valueOf(probe.getRemainingTokens()));
            chain.doFilter(request, response);
            return;
        }
        long waitSeconds = Math.max(1, TimeUnit.NANOSECONDS.toSeconds(probe.getNanosToWaitForRefill()));
        response.setStatus(429);
        response.setHeader("Retry-After", String.valueOf(waitSeconds));
    }
}

Advanced An in-memory limiter is per instance: with 5 replicas a limit of 10/s becomes 50/s. For a global limit keep the counters in Redis (Bucket4j's Redis integrations, or Spring Cloud Gateway's RequestRateLimiter, which runs a token bucket as an atomic Redis script). Sliding-window counters need two keys per client per window and an atomic increment-and-read.

Trade-offs

BenefitsCosts
Fixed windowTrivial: one counter per client.Boundary bursts of up to 2× the limit.
Sliding windowAccurate without storing every timestamp.An approximation; slightly more state.
Token bucketExplicit burst allowance, easy to explain to API users.Bursts still reach the backend (up to the bucket size).
Leaky bucketSmooth, predictable load downstream.Adds latency; a full queue still drops requests.

Production considerations

  • Return 429 Too Many Requests with Retry-After and rate-limit headers so well-behaved clients slow down instead of retrying blindly (see Retry).
  • Limit per identity (API key, user) rather than per IP where possible; NAT puts many users behind one address.
  • Layer limits: a coarse global limit at the edge, finer per-endpoint limits in services.
  • Exempt health checks and internal traffic, and alert on clients that are constantly throttled.

Further reading

Related topics