Skip to content
System Design Lab
Real Problems

Orders Without Invoices: Lost Events After Commit

Orders were saved but Billing never heard about some of them. The culprit: committing to the database and publishing to the broker as two separate steps. Reproduce it, then fix it with an outbox.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Recognise the dual-write problem behind silently missing events.
  • Reproduce the loss with broker outages and badly timed restarts.
  • Fix it with a transactional outbox, using either a polling relay or CDC.
  • Detect lost events with reconciliation before customers do.

Before you start

Symptoms

  • Finance finds paid orders with no invoice — roughly a few per thousand, clustered around certain times.
  • Order Service logs show TimeoutException / NetworkException from the Kafka producer, or nothing at all.
  • The gaps line up with broker maintenance windows and with deployments.
  • Billing has no errors: it never received the events.

System context

CustomersCustomersOrder Servicesave, then publishOrder ServiceOrders DBOrders DBorder-eventsorder-eventsBillingBilling
Order Service saves the order, then publishes OrderPlaced to Kafka; Billing creates invoices from the events.

Root cause

The bug: two writes, no shared transaction
@Transactional
public void placeOrder(Order order) {
    orders.save(order);
}                                         // ← commit happens here

public void checkout(Order order) {
    placeOrder(order);
    // A broker outage, a timeout or a process restart right here loses the event forever.
    kafka.send("order-events", order.id(), new OrderPlaced(order));
}
  1. The database commit and the broker publish are two separate operations; there is no transaction spanning both.
  2. If the broker is unreachable, the send fails after the order is already committed. Retrying in memory only helps until the process dies.
  3. If the process restarts (deploy, OOM kill, node drain) between the two steps, the event is never sent and nobody notices.

Interactive reproduction

The broker is down at t = 8–12 s and 22–24 s, and 2% of writes hit a restart. In A watch Orders with no event (lost) climb. B and C write the event to an outbox in the same transaction: nothing is lost, the backlog simply drains after each outage.

Loading simulation…

Solution alternatives

OptionLost eventsNotes
Retry the publish in memoryStill lost on restartNarrows the window, does not close it.
Distributed transaction (XA) across DB and brokerNone in theoryFew brokers support it well; slow and fragile (see 2PC failures).
Transactional outbox + polling relayNoneSimple; publish latency ≈ poll interval; adds query load.
Transactional outbox + CDC (Debezium)NoneLow latency, no polling; needs a connector to operate.
Event sourcing with store subscriptionsNoneThe event *is* the write; a big modelling change.

Recommended fix

Write the event to an outbox table in the same transaction as the order, and let a separate relay publish it. See Transactional Outbox for the full pattern.

One transaction, two rows
@Transactional
public void checkout(Order order) {
    orders.save(order);
    outbox.save(OutboxEvent.of("Order", order.id(), "OrderPlaced", new OrderPlaced(order)));
}   // both rows commit — or neither does

Advanced The relay publishes at least once: after a crash it may resend an event it already sent. That is why Billing in the simulation is idempotent — see Idempotent Consumer. Use the event id from the outbox row as the deduplication key.

Verification strategy

  • Chaos test: stop the broker for a minute during a load test; the number of invoices must equal the number of orders afterwards.
  • Kill the Order Service pod repeatedly during load (e.g. every 30 s) and compare counts again.
  • Assert the outbox backlog returns to zero after each outage.

Monitoring and prevention

  • Reconciliation job: daily, count orders vs invoices per hour; any gap pages someone.
  • Outbox backlog and oldest-unsent age as metrics with alerts.
  • Code review rule: a kafka.send after a @Transactional boundary is a red flag.

Lessons learned

  • Two writes to two systems is a distributed transaction, whether you call it one or not.
  • Lost messages are silent: you only find them by reconciling, never from error logs.
  • Make the database the single point of commit, and publish from it.

Further reading

Related topics