Microservice Patterns
Circuit Breaker
Stop calling a dependency that is failing, fail fast instead, and probe carefully until it recovers — so one broken service cannot drag down everything that calls it.
What you will learn
- Explain why calling a failing dependency again and again hurts the caller more than the dependency.
- Trace the CLOSED → OPEN → HALF_OPEN → CLOSED state machine and what triggers each transition.
- Tune failure threshold, sliding window, minimum calls and wait duration — and predict the effect.
- Configure a Resilience4j circuit breaker with a fallback in Spring Boot.
Before you start
Overview
Think of the breaker in your home's fuse box. When a faulty appliance draws too much current, the breaker trips and cuts the circuit so the wiring does not catch fire. You fix the appliance, flip the breaker back, and power returns. A software circuit breaker does the same for network calls: when a dependency keeps failing, it stops sending calls to it for a while.
A circuit breaker is a small proxy that sits in front of a remote call. It counts recent successes and failures. While things are healthy it lets every call through. When the failure rate crosses a threshold it opens: further calls fail immediately without touching the network. After a cool-down it lets a few probe calls through to check whether the dependency has recovered.
The problem: failing slowly is worse than failing fast
Imagine an order service that calls a payment service on every checkout. The payment service starts timing out. Every checkout request now waits for the full timeout before failing. Each waiting request holds a thread, a database connection and memory in the order service. Within seconds the order service has no free threads — and now requests that never needed payment (browsing, order history) fail too. One sick dependency has taken down a healthy service.
- Resource exhaustion — thread pools, connection pools and in-flight request limits fill up with calls that are waiting on a dependency that will not answer.
- Latency amplification — the caller's p99 becomes the dependency's timeout, and every upstream service inherits it.
- Retry storms — clients and intermediate services retry, multiplying load on the exact component that is already overloaded. See Retry.
- Slower recovery — a struggling service that keeps receiving full traffic rarely gets the breathing room it needs to recover.
The solution: a state machine around the call
The breaker wraps the call and keeps a sliding window of the most recent outcomes. It moves between three states:
| State | What happens to a call | Leaves the state when |
|---|---|---|
| CLOSED | Forwarded to the dependency. The outcome is recorded in the sliding window. | The window holds at least the minimum number of calls and the failure rate is at or above the threshold → OPEN. |
| OPEN | Rejected immediately with an error (or a fallback). The dependency is not called at all. | The wait duration elapses → HALF_OPEN. |
| HALF_OPEN | A small, fixed number of probe calls are forwarded; any extra calls are rejected. | All probes finished: failure rate below threshold → CLOSED, otherwise → OPEN again. |
Advanced Production libraries offer count-based windows (the last N calls) and time-based windows (calls in the last N seconds). Count-based windows react quickly at low traffic; time-based windows are steadier at high traffic. minimumNumberOfCalls stops the breaker from opening after one unlucky call at startup. Many libraries also track a separate slow-call rate, so a dependency that answers successfully but takes 5 seconds can still open the circuit.
Architecture
Step through the lifecycle
Press play or step through the interaction.
Run it: five scenarios
Each scenario below runs on the same deterministic engine. Press Play, watch the packets, then open the State machine and Timeline tabs. Try injecting a failure yourself with Payment down — or change the threshold while it runs.
Implementation with Spring Boot and Resilience4j
With the resilience4j-spring-boot3 starter (plus spring-boot-starter-aop), a breaker is configured per dependency in application.yml and attached with an annotation.
resilience4j:
circuitbreaker:
instances:
payment:
sliding-window-type: COUNT_BASED
sliding-window-size: 10
minimum-number-of-calls: 5
failure-rate-threshold: 50
wait-duration-in-open-state: 5s
permitted-number-of-calls-in-half-open-state: 3
automatic-transition-from-open-to-half-open-enabled: true
slow-call-duration-threshold: 2s
slow-call-rate-threshold: 100
record-exceptions:
- java.io.IOException
- java.util.concurrent.TimeoutException
- org.springframework.web.client.HttpServerErrorException
ignore-exceptions:
- com.example.payment.CardDeclinedException@Service
public class PaymentClient {
private final RestClient restClient;
public PaymentClient(RestClient.Builder builder) {
this.restClient = builder.baseUrl("http://payment-service").build();
}
@CircuitBreaker(name = "payment", fallbackMethod = "chargeFallback")
public PaymentResult charge(ChargeRequest request) {
return restClient.post()
.uri("/charges")
.body(request)
.retrieve()
.body(PaymentResult.class);
}
// Circuit is OPEN (or HALF_OPEN and out of probes): accept the order, charge later.
private PaymentResult chargeFallback(ChargeRequest request, CallNotPermittedException ex) {
return PaymentResult.pending(request.orderId());
}
// The call itself failed while the circuit was CLOSED.
private PaymentResult chargeFallback(ChargeRequest request, Exception ex) {
throw new PaymentUnavailableException(request.orderId(), ex);
}
}Advanced Resilience4j picks the fallback whose exception parameter is the closest match, so you can treat "circuit open" differently from "call failed". Business errors such as a declined card are listed in ignore-exceptions: they are valid answers, not signs of an unhealthy dependency, and must not trip the breaker.
@Configuration
class CircuitBreakerLogging {
CircuitBreakerLogging(CircuitBreakerRegistry registry) {
registry.circuitBreaker("payment")
.getEventPublisher()
.onStateTransition(event -> log.warn("payment circuit {}", event.getStateTransition()))
.onCallNotPermitted(event -> log.debug("payment call rejected (circuit open)"));
}
private static final Logger log = LoggerFactory.getLogger(CircuitBreakerLogging.class);
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Availability | Healthy parts of the caller keep working while a dependency is down. | Some requests that might have succeeded are rejected while the circuit is OPEN. |
| Latency | Failures become fast (microseconds instead of a full timeout). | None while CLOSED, apart from tiny bookkeeping overhead. |
| Recovery | The dependency gets breathing room; probes limit load while it warms up. | Recovery is noticed only after the wait duration, never instantly. |
| Operations | State changes are clear, alertable signals. | Thresholds need tuning per dependency, and a badly tuned breaker flaps between states. |
Production considerations
- One breaker per dependency (and sometimes per endpoint). A single shared breaker lets a broken report endpoint block the healthy checkout endpoint.
- Combine with timeouts. A breaker only sees failures; without a Timeout a hanging call is never recorded as one.
- Decide on the fallback with the product team. Cached data, a default, a queued "we'll process this later" or a clear error — each is a product decision.
- Do not count client errors. 4xx responses and business rejections mean the dependency is healthy.
- Order of decorators matters. Resilience4j's default aspect order is Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead → call. Retries outside the breaker stop instantly once it opens, because the breaker rejects immediately.
- Breakers are per instance. Ten replicas means ten independent breakers; each opens on its own sample of traffic. That is usually fine and avoids a shared coordination point.
- Low traffic — with one call a minute a count-based window of 100 may take hours to fill; use a smaller window or a time-based one with a sensible minimum.