Microservice Patterns
Transactional Outbox
Save a business change and the event announcing it in the same database transaction, then publish the event from that table — so you never lose an event or publish one for a change that rolled back.
What you will learn
- Explain the dual-write problem and why reordering the two writes does not fix it.
- Describe the outbox table and the relay (polling publisher or change data capture).
- Understand why the outbox gives at-least-once delivery, not exactly-once.
- Implement an outbox with Spring Data and a scheduled relay.
Overview
A shop records each sale in its ledger and also has to phone the warehouse. If the clerk writes the sale and then gets interrupted before phoning, the parcel never ships. The fix: write "phone the warehouse about sale #42" on a to-do list in the same ledger, on the same line. Someone works through the to-do list and ticks items off. That to-do list is the outbox.
In event-driven systems a service often has to do two things together: update its database and publish an event to a broker such as Kafka. The database and the broker cannot share a transaction. The outbox pattern turns the event into a row in an outbox table, written in the same local transaction as the business change. A separate relay reads the table and publishes the events.
The problem: dual writes
Press play or step through the interaction.
- Save, then publish — a crash or broker outage in between leaves an order no other service hears about.
- Publish, then save — if the database transaction then fails, everyone hears about an order that does not exist.
- Publish inside the transaction — the broker message cannot be rolled back, and a slow broker now holds database locks.
Run it: lost events vs a backlog
Both scenarios have the same crashes and the same 6-second broker outage. With dual writes, Lost events climbs and never comes back. With the outbox, the Outbox backlog grows during the outage and drains afterwards — nothing is lost, only delayed.
Two ways to run the relay
| Polling publisher | Change data capture (CDC) | |
|---|---|---|
| How | A scheduled job selects unsent rows, publishes, marks them sent. | A tool such as Debezium tails the database log and publishes inserted outbox rows. |
| Latency | Poll interval (e.g. 100 ms – 1 s). | Near real time. |
| Database load | Extra queries; needs an index on the unsent rows. | Reads the replication log; no query load. |
| Operations | Just your application code. | Kafka Connect + Debezium to run and monitor. |
| Ordering | Preserved per aggregate if you publish in insertion order. | Preserved per aggregate via the log order and message key. |
Implementation with Spring
CREATE TABLE outbox (
id UUID PRIMARY KEY,
aggregate_type VARCHAR(64) NOT NULL,
aggregate_id VARCHAR(64) NOT NULL, -- becomes the Kafka key: keeps per-order ordering
event_type VARCHAR(128) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;@Service
public class OrderService {
private final OrderRepository orders;
private final OutboxRepository outbox;
OrderService(OrderRepository orders, OutboxRepository outbox) {
this.orders = orders;
this.outbox = outbox;
}
@Transactional // one local transaction: both rows commit, or neither does
public Order placeOrder(PlaceOrder command) {
Order order = orders.save(Order.place(command));
outbox.save(OutboxEvent.of("Order", order.id(), "OrderPlaced", OrderPlaced.from(order)));
return order;
}
}@Component
class OutboxRelay {
private final OutboxRepository outbox;
private final KafkaTemplate<String, String> kafka;
OutboxRelay(OutboxRepository outbox, KafkaTemplate<String, String> kafka) {
this.outbox = outbox;
this.kafka = kafka;
}
@Scheduled(fixedDelay = 200)
@Transactional
public void publishPending() {
// SELECT ... WHERE published_at IS NULL ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED
for (OutboxEvent event : outbox.lockNextBatch(100)) {
kafka.send("order-events", event.aggregateId(), event.payload()).join(); // wait for the ack
event.markPublished(); // a crash before commit means this event is sent again
}
}
}Advanced FOR UPDATE SKIP LOCKED lets several relay instances run without publishing the same row concurrently. Publishing synchronously (waiting for the broker ack) before marking the row keeps the at-least-once guarantee. Delete or archive published rows regularly so the table stays small.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Reliability | No lost events and no events for rolled-back changes. | Duplicates are possible: consumers must be idempotent. |
| Latency | The user's request does not wait for the broker. | Events arrive after the poll interval (or CDC lag). |
| Operations | Survives broker outages gracefully with a growing backlog. | An extra table, a relay to run, cleanup of old rows. |
Production considerations
- Alert on outbox backlog size and on the age of the oldest unpublished row.
- Use the aggregate id as the message key so events for one order stay in order.
- Put an event id in the payload — consumers use it for deduplication (see Idempotent Consumer).