Skip to content
System Design Lab
Microservice Patterns

Pipes and Filters

Break a multi-step processing job into independent filters connected by queues (pipes), so each step can be scaled, deployed and diagnosed on its own — and a slow step is visible as a growing queue in front of it.

Beginner 4 min read· Updated 2026-11-18

What you will learn

  • Decompose a processing job into filters and pipes.
  • See how a pipeline localises a bottleneck to one stage.
  • Scale only the stage that needs it.
  • Handle failures, retries and ordering between stages.

Before you start

Overview

A car factory does not have each worker build a whole car. It has stations — frame, paint, engine, inspection — with parts waiting between them. If painting is slow, you see cars piling up before the paint shop and add a second paint line, not a second factory.

With pipes and filters, each processing step is a filter: a small, independent component that reads input, does one thing and writes output. Filters are connected by pipes — usually queues or topics. Because stages are independent, each can run with its own number of workers, technology and release cycle, and the queue depth in front of each stage tells you where the work is waiting.

Run it

Images go through decode → resize → watermark → publish. At t = 10 s a new camera format makes resizing three times slower. Compare where the backlog appears and what it takes to clear it.

Loading simulation…

Designing filters

  • One responsibility per filter, with a clear input and output contract (a schema per pipe).
  • Idempotent filters: queues redeliver, so processing the same item twice must be harmless.
  • Pass references for big data (claim check): the pipe carries the image location, not the image.
  • Dead-letter per stage so one bad item does not block the pipeline (dead letter queue).
  • Ordering: parallel workers reorder items; keep ordering only where a stage really needs it.

A filter

Read from one queue, write to the next
@RabbitListener(queues = "images.resize", concurrency = "5")   // scale this stage on its own
void resize(ImageRef in) {
    BufferedImage image = storage.read(in.location());
    String out = storage.write(in.id() + "-1200.webp", Thumbnails.of(image).width(1200).asBufferedImage());
    rabbit.convertAndSend("images.watermark", new ImageRef(in.id(), out));   // next pipe
}

Trade-offs

BenefitsCosts
ScalingScale the slow stage only; cheap stages stay small.Each stage needs capacity planning and monitoring.
ChangeAdd, replace or reorder filters independently.Contracts between stages must be versioned.
LatencyThroughput scales and failures stay local.Every pipe adds queueing delay and serialisation; end-to-end tracing needs effort.

Production considerations

  • Dashboard queue depth and age for every pipe — it is the map of your bottlenecks.
  • Autoscale each stage on its own queue depth.
  • Propagate a correlation id through every stage so one item can be traced end to end.
  • For very simple, fast pipelines, an in-process chain of functions may be enough.

Further reading

Related topics