Microservice Patterns
Claim Check
Keep large payloads out of the message bus: store them in blob storage and send only a small token that points to them, so the broker stays fast, cheap and within its size limits.
What you will learn
- Explain why large messages hurt brokers.
- Apply the claim-check flow: store, send token, fetch.
- Handle payload lifecycle, security and failures.
- Know when inline payloads are still fine.
Before you start
Overview
At a cloakroom you hand over your heavy coat and get a small numbered ticket. You carry the ticket, not the coat; whoever has the ticket can collect the coat later.
Brokers are built for many small messages. Large payloads — scans, videos, big JSON documents — hit size limits (often 256 KB–1 MB by default), slow replication and consume memory and network on every hop. With a claim check, the producer stores the payload in object storage and publishes only a small message containing its location; consumers fetch the payload when they need it.
Run it
Scanners upload 5 MB documents. Compare Message bus busy and latency when the payload travels through the bus (A) and when only a claim token does (B). Raise the traffic in both.
Implementation
public void submit(Scan scan) {
String key = "scans/" + scan.id() + ".pdf";
s3.putObject(PutObjectRequest.builder().bucket("scans").key(key).build(), RequestBody.fromBytes(scan.bytes()));
kafka.send("scans", scan.id(), new ScanUploaded(scan.id(), "s3://scans/" + key, scan.bytes().length, sha256(scan.bytes())));
}
@KafkaListener(topics = "scans", groupId = "ocr")
void on(ScanUploaded event) {
byte[] pdf = s3.getObjectAsBytes(b -> b.bucket("scans").key(keyOf(event.location()))).asByteArray();
verifyChecksum(pdf, event.sha256());
ocr.process(event.scanId(), pdf);
}- Order of writes: store the payload first, then publish — a token must never point to nothing.
- Lifecycle: delete or expire payloads when every consumer is done (storage lifecycle rules, or a TTL longer than the topic's retention).
- Security: consumers need read access to the storage; use short-lived signed URLs (valet key) when the consumer is external.
- Integrity: include size and checksum in the token.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Broker | Small, fast messages; within size limits; cheaper retention and replication. | The broker no longer guarantees the payload exists — storage does. |
| Latency | Bus latency stays low under heavy payload traffic. | An extra storage round trip to write and to read. |
| Operations | Object storage scales cheaply for big data. | Two systems to secure, monitor and clean up consistently. |
Production considerations
- Monitor orphaned payloads (stored but never referenced) and dangling tokens (referenced but missing).
- Keep personal data in storage with encryption and access logging; the token itself should reveal nothing.
- Replaying old messages requires the payloads to still exist — align retention.
- Consumers that only need metadata can skip fetching the payload entirely.