Microservice Patterns
Leader Election
When work must be done by exactly one of many identical instances — a scheduled job, a coordinator, a single writer — let them elect a leader with an expiring lease, and fence out a leader that comes back from the dead.
What you will learn
- Recognise work that must not run on every replica.
- Implement lease-based leader election and understand failover time.
- Explain split brain and how fencing tokens prevent it.
- Choose a coordination service (database row, etcd, ZooKeeper, Kubernetes Lease).
Before you start
Overview
In a shared flat, whoever holds the key to the post box empties it — not everyone. The key comes with a rule: if you are away for a week, you hand it on. Otherwise the post either piles up or gets opened three times.
Horizontal scaling gives you identical replicas, but some work must happen exactly once: charging subscriptions, sending a daily report, being the single writer to a partition. Leader election lets replicas agree that one of them is in charge. The usual mechanism is a lease in a strongly consistent store: the holder must renew it before it expires; if it stops renewing (crash, partition), another replica acquires it.
Run it
Three billing replicas each have a scheduler. Compare Periods run more than once with and without a lease, watch the gap after billing-1 crashes at t = 12 s, and change the lease length.
Split brain and fencing
A leader can pause (GC, VM migration) longer than its lease, wake up still believing it leads, and act alongside the new leader. Every lease acquisition therefore returns a fencing token — a number that only ever increases — and every protected resource rejects requests with a token lower than the highest it has seen.
-- Acquire or renew: succeeds only if the lease is free, expired, or already ours.
UPDATE leases
SET holder = :me, expires_at = now() + interval '10 seconds',
token = CASE WHEN holder = :me THEN token ELSE token + 1 END
WHERE name = 'billing' AND (holder = :me OR expires_at < now())
RETURNING token;
-- Every write by the leader carries its token; stale leaders are rejected.
UPDATE billing_runs SET done = true, token = :token
WHERE period = :period AND token <= :token;@Scheduled(cron = "0 * * * * *")
@SchedulerLock(name = "chargeSubscriptions", lockAtMostFor = "50s", lockAtLeastFor = "5s")
void chargeSubscriptions() {
billing.chargeDue(); // runs on one replica per minute; still make it idempotent
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Correctness | Exactly-one execution for work that must not be duplicated. | Needs fencing (or idempotent work) to be safe against paused leaders. |
| Availability | Automatic takeover when the leader dies. | Nobody does the work for up to one lease period after a crash. |
| Dependencies | Simple with a database row or a Kubernetes Lease. | The coordination store becomes critical; it must be strongly consistent. |
Production considerations
- Prefer making the work idempotent and partitioned; use a leader only when you must.
- Choose the lease length from acceptable failover time and worst-case pauses; renew at a third of it.
- Expose who the leader is (metric or health detail) and alert if there is none for too long.
- Never use an eventually consistent cache as the lock store.