Microservice Patterns
Pipes and Filters
Break a multi-step processing job into independent filters connected by queues (pipes), so each step can be scaled, deployed and diagnosed on its own — and a slow step is visible as a growing queue in front of it.
What you will learn
- Decompose a processing job into filters and pipes.
- See how a pipeline localises a bottleneck to one stage.
- Scale only the stage that needs it.
- Handle failures, retries and ordering between stages.
Before you start
Overview
A car factory does not have each worker build a whole car. It has stations — frame, paint, engine, inspection — with parts waiting between them. If painting is slow, you see cars piling up before the paint shop and add a second paint line, not a second factory.
With pipes and filters, each processing step is a filter: a small, independent component that reads input, does one thing and writes output. Filters are connected by pipes — usually queues or topics. Because stages are independent, each can run with its own number of workers, technology and release cycle, and the queue depth in front of each stage tells you where the work is waiting.
Run it
Images go through decode → resize → watermark → publish. At t = 10 s a new camera format makes resizing three times slower. Compare where the backlog appears and what it takes to clear it.
Designing filters
- One responsibility per filter, with a clear input and output contract (a schema per pipe).
- Idempotent filters: queues redeliver, so processing the same item twice must be harmless.
- Pass references for big data (claim check): the pipe carries the image location, not the image.
- Dead-letter per stage so one bad item does not block the pipeline (dead letter queue).
- Ordering: parallel workers reorder items; keep ordering only where a stage really needs it.
A filter
@RabbitListener(queues = "images.resize", concurrency = "5") // scale this stage on its own
void resize(ImageRef in) {
BufferedImage image = storage.read(in.location());
String out = storage.write(in.id() + "-1200.webp", Thumbnails.of(image).width(1200).asBufferedImage());
rabbit.convertAndSend("images.watermark", new ImageRef(in.id(), out)); // next pipe
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Scaling | Scale the slow stage only; cheap stages stay small. | Each stage needs capacity planning and monitoring. |
| Change | Add, replace or reorder filters independently. | Contracts between stages must be versioned. |
| Latency | Throughput scales and failures stay local. | Every pipe adds queueing delay and serialisation; end-to-end tracing needs effort. |
Production considerations
- Dashboard queue depth and age for every pipe — it is the map of your bottlenecks.
- Autoscale each stage on its own queue depth.
- Propagate a correlation id through every stage so one item can be traced end to end.
- For very simple, fast pipelines, an in-process chain of functions may be enough.