Skip to content
System Design Lab
Microservice Patterns

Scheduler Agent Supervisor

Run a multi-step distributed workflow with three roles: a scheduler that drives the steps, agents that do the remote work, and a supervisor that finds steps that stalled or failed and retries or compensates them.

Advanced 4 min read· Updated 2026-11-18

What you will learn

  • Describe the three roles and the durable state they share.
  • See how lost replies leave workflows stuck forever without supervision.
  • Recover stuck steps with deadlines, retries and compensation.
  • Relate the pattern to sagas and workflow engines.

Before you start

Overview

A wedding planner (scheduler) books the venue, the caterer and the band (agents). A good planner also keeps a checklist and calls back anyone who has not confirmed by the deadline (supervisor) — otherwise one lost email means no band on the day.

The scheduler starts each step of a workflow and records progress in a durable state store. Agents wrap the remote work (a payment provider, a booking API) and report results. Because messages can be lost and agents can crash, a separate supervisor periodically scans the state store for steps that missed their deadline or failed, and retries them (agents must be idempotent) or triggers compensation.

Run it

Bookings run three steps. Between t = 8 s and 14 s the network to charge-card silently drops messages. Watch Bookings in progress after the network recovers, with and without the supervisor.

Loading simulation…

The state store

FieldPurpose
workflow_id, stepWhere the workflow is.
status (pending, running, done, failed)What happened last.
deadlineWhen the supervisor should consider the step lost.
attemptsHow many retries are left before compensating.
lock_owner, versionSo only one scheduler/supervisor acts on a workflow at a time.

Supervisor sketch

Find steps past their deadline and recover them
@Scheduled(fixedDelay = 1000)
@SchedulerLock(name = "booking-supervisor")          // one supervisor at a time
void supervise() {
    for (WorkflowStep step : steps.findRunningPastDeadline(Instant.now(), 100)) {
        if (step.attempts() < MAX_ATTEMPTS) {
            steps.markRetrying(step.id(), step.version(), Instant.now().plus(STEP_TIMEOUT));
            agents.resend(step);                       // agents are idempotent: safe to repeat
        } else {
            steps.markFailed(step.id(), step.version());
            compensations.start(step.workflowId());    // undo completed steps, like a saga
        }
    }
}

Trade-offs

BenefitsCosts
ResilienceLost messages and crashed agents no longer leave work stuck.Retries require idempotent agents; recovery happens at the scan interval.
VisibilityThe state store shows every workflow's progress.The store and supervisor are critical components to run and scale.
ComplexityClear separation of driving, doing and checking.More moving parts than a simple synchronous call chain.

Production considerations

  • Set deadlines per step from the agent's real latency, not one global value.
  • Alert on workflows stuck longer than expected and on compensation rate.
  • Make the supervisor itself highly available (a lease, see leader election).
  • Record every attempt — when a workflow goes wrong, the history is what support needs.

Further reading

Related topics