Skip to content
System Design Lab
Microservice Patterns

Leader Election

When work must be done by exactly one of many identical instances — a scheduled job, a coordinator, a single writer — let them elect a leader with an expiring lease, and fence out a leader that comes back from the dead.

Advanced 4 min read· Updated 2026-11-18

What you will learn

  • Recognise work that must not run on every replica.
  • Implement lease-based leader election and understand failover time.
  • Explain split brain and how fencing tokens prevent it.
  • Choose a coordination service (database row, etcd, ZooKeeper, Kubernetes Lease).

Before you start

Overview

In a shared flat, whoever holds the key to the post box empties it — not everyone. The key comes with a rule: if you are away for a week, you hand it on. Otherwise the post either piles up or gets opened three times.

Horizontal scaling gives you identical replicas, but some work must happen exactly once: charging subscriptions, sending a daily report, being the single writer to a partition. Leader election lets replicas agree that one of them is in charge. The usual mechanism is a lease in a strongly consistent store: the holder must renew it before it expires; if it stops renewing (crash, partition), another replica acquires it.

Run it

Three billing replicas each have a scheduler. Compare Periods run more than once with and without a lease, watch the gap after billing-1 crashes at t = 12 s, and change the lease length.

Loading simulation…

Split brain and fencing

A leader can pause (GC, VM migration) longer than its lease, wake up still believing it leads, and act alongside the new leader. Every lease acquisition therefore returns a fencing token — a number that only ever increases — and every protected resource rejects requests with a token lower than the highest it has seen.

A lease in a database row, with a fencing token
-- Acquire or renew: succeeds only if the lease is free, expired, or already ours.
UPDATE leases
   SET holder = :me, expires_at = now() + interval '10 seconds',
       token = CASE WHEN holder = :me THEN token ELSE token + 1 END
 WHERE name = 'billing' AND (holder = :me OR expires_at < now())
RETURNING token;

-- Every write by the leader carries its token; stale leaders are rejected.
UPDATE billing_runs SET done = true, token = :token
 WHERE period = :period AND token <= :token;
Simplest option for scheduled jobs: ShedLock
@Scheduled(cron = "0 * * * * *")
@SchedulerLock(name = "chargeSubscriptions", lockAtMostFor = "50s", lockAtLeastFor = "5s")
void chargeSubscriptions() {
    billing.chargeDue();     // runs on one replica per minute; still make it idempotent
}

Trade-offs

BenefitsCosts
CorrectnessExactly-one execution for work that must not be duplicated.Needs fencing (or idempotent work) to be safe against paused leaders.
AvailabilityAutomatic takeover when the leader dies.Nobody does the work for up to one lease period after a crash.
DependenciesSimple with a database row or a Kubernetes Lease.The coordination store becomes critical; it must be strongly consistent.

Production considerations

  • Prefer making the work idempotent and partitioned; use a leader only when you must.
  • Choose the lease length from acceptable failover time and worst-case pauses; renew at a third of it.
  • Expose who the leader is (metric or health detail) and alert if there is none for too long.
  • Never use an eventually consistent cache as the lock store.

Further reading

Related topics