Microservice Patterns
Scheduler Agent Supervisor
Run a multi-step distributed workflow with three roles: a scheduler that drives the steps, agents that do the remote work, and a supervisor that finds steps that stalled or failed and retries or compensates them.
What you will learn
- Describe the three roles and the durable state they share.
- See how lost replies leave workflows stuck forever without supervision.
- Recover stuck steps with deadlines, retries and compensation.
- Relate the pattern to sagas and workflow engines.
Before you start
Overview
A wedding planner (scheduler) books the venue, the caterer and the band (agents). A good planner also keeps a checklist and calls back anyone who has not confirmed by the deadline (supervisor) — otherwise one lost email means no band on the day.
The scheduler starts each step of a workflow and records progress in a durable state store. Agents wrap the remote work (a payment provider, a booking API) and report results. Because messages can be lost and agents can crash, a separate supervisor periodically scans the state store for steps that missed their deadline or failed, and retries them (agents must be idempotent) or triggers compensation.
Run it
Bookings run three steps. Between t = 8 s and 14 s the network to charge-card silently drops messages. Watch Bookings in progress after the network recovers, with and without the supervisor.
The state store
| Field | Purpose |
|---|---|
workflow_id, step | Where the workflow is. |
status (pending, running, done, failed) | What happened last. |
deadline | When the supervisor should consider the step lost. |
attempts | How many retries are left before compensating. |
lock_owner, version | So only one scheduler/supervisor acts on a workflow at a time. |
Supervisor sketch
@Scheduled(fixedDelay = 1000)
@SchedulerLock(name = "booking-supervisor") // one supervisor at a time
void supervise() {
for (WorkflowStep step : steps.findRunningPastDeadline(Instant.now(), 100)) {
if (step.attempts() < MAX_ATTEMPTS) {
steps.markRetrying(step.id(), step.version(), Instant.now().plus(STEP_TIMEOUT));
agents.resend(step); // agents are idempotent: safe to repeat
} else {
steps.markFailed(step.id(), step.version());
compensations.start(step.workflowId()); // undo completed steps, like a saga
}
}
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Resilience | Lost messages and crashed agents no longer leave work stuck. | Retries require idempotent agents; recovery happens at the scan interval. |
| Visibility | The state store shows every workflow's progress. | The store and supervisor are critical components to run and scale. |
| Complexity | Clear separation of driving, doing and checking. | More moving parts than a simple synchronous call chain. |
Production considerations
- Set deadlines per step from the agent's real latency, not one global value.
- Alert on workflows stuck longer than expected and on compensation rate.
- Make the supervisor itself highly available (a lease, see leader election).
- Record every attempt — when a workflow goes wrong, the history is what support needs.