Real Problems
Orders Without Invoices: Lost Events After Commit
Orders were saved but Billing never heard about some of them. The culprit: committing to the database and publishing to the broker as two separate steps. Reproduce it, then fix it with an outbox.
What you will learn
- Recognise the dual-write problem behind silently missing events.
- Reproduce the loss with broker outages and badly timed restarts.
- Fix it with a transactional outbox, using either a polling relay or CDC.
- Detect lost events with reconciliation before customers do.
Before you start
Symptoms
- Finance finds paid orders with no invoice — roughly a few per thousand, clustered around certain times.
- Order Service logs show
TimeoutException/NetworkExceptionfrom the Kafka producer, or nothing at all. - The gaps line up with broker maintenance windows and with deployments.
- Billing has no errors: it never received the events.
System context
Root cause
@Transactional
public void placeOrder(Order order) {
orders.save(order);
} // ← commit happens here
public void checkout(Order order) {
placeOrder(order);
// A broker outage, a timeout or a process restart right here loses the event forever.
kafka.send("order-events", order.id(), new OrderPlaced(order));
}- The database commit and the broker publish are two separate operations; there is no transaction spanning both.
- If the broker is unreachable, the send fails after the order is already committed. Retrying in memory only helps until the process dies.
- If the process restarts (deploy, OOM kill, node drain) between the two steps, the event is never sent and nobody notices.
Interactive reproduction
The broker is down at t = 8–12 s and 22–24 s, and 2% of writes hit a restart. In A watch Orders with no event (lost) climb. B and C write the event to an outbox in the same transaction: nothing is lost, the backlog simply drains after each outage.
Solution alternatives
| Option | Lost events | Notes |
|---|---|---|
| Retry the publish in memory | Still lost on restart | Narrows the window, does not close it. |
| Distributed transaction (XA) across DB and broker | None in theory | Few brokers support it well; slow and fragile (see 2PC failures). |
| Transactional outbox + polling relay | None | Simple; publish latency ≈ poll interval; adds query load. |
| Transactional outbox + CDC (Debezium) | None | Low latency, no polling; needs a connector to operate. |
| Event sourcing with store subscriptions | None | The event *is* the write; a big modelling change. |
Recommended fix
Write the event to an outbox table in the same transaction as the order, and let a separate relay publish it. See Transactional Outbox for the full pattern.
@Transactional
public void checkout(Order order) {
orders.save(order);
outbox.save(OutboxEvent.of("Order", order.id(), "OrderPlaced", new OrderPlaced(order)));
} // both rows commit — or neither doesAdvanced The relay publishes at least once: after a crash it may resend an event it already sent. That is why Billing in the simulation is idempotent — see Idempotent Consumer. Use the event id from the outbox row as the deduplication key.
Verification strategy
- Chaos test: stop the broker for a minute during a load test; the number of invoices must equal the number of orders afterwards.
- Kill the Order Service pod repeatedly during load (e.g. every 30 s) and compare counts again.
- Assert the outbox backlog returns to zero after each outage.
Monitoring and prevention
- Reconciliation job: daily, count orders vs invoices per hour; any gap pages someone.
- Outbox backlog and oldest-unsent age as metrics with alerts.
- Code review rule: a
kafka.sendafter a@Transactionalboundary is a red flag.
Lessons learned
- Two writes to two systems is a distributed transaction, whether you call it one or not.
- Lost messages are silent: you only find them by reconciling, never from error logs.
- Make the database the single point of commit, and publish from it.