Microservice Patterns
Distributed Tracing
Follow one request through every service it touches as a tree of timed spans, so you can see exactly which hop made it slow or failed — something averages and per-service logs cannot show.
What you will learn
- Explain traces, spans and context propagation.
- Read a trace waterfall: nesting, sequential vs parallel calls, gaps.
- Find a slow dependency and an N+1 query from traces.
- Instrument services with OpenTelemetry and choose a sampling strategy.
Before you start
Overview
A parcel tracking page shows every stop a parcel made and how long it waited at each depot. If it arrived late, you can see that it sat in one warehouse for two days. A trace is the same for a request: every service it passed through, and how long each took.
A trace is one request's journey. Each unit of work — handling an HTTP request, a database query, a call to another service — is a span with a start, a duration, a status and a parent. Services pass the trace id and parent span id along with every call (context propagation, standardised as the W3C traceparent header), so a tracing backend can rebuild the tree.
Run it
Open the Trace tab while the simulation runs. A shows a healthy checkout. In B, p99 is high but nothing errors — pick *Slowest recent* to find the culprit. C hides an N+1 query.
Reading a waterfall
- Width is duration; the widest leaf span is usually the cause, its ancestors are only waiting.
- Staircases of many short spans to the same component mean a loop of calls (N+1 queries).
- Spans one after another that could run in parallel are free latency savings.
- Gaps between a parent's start and its children are time spent inside the parent (CPU, locks, queueing).
Instrumentation with OpenTelemetry
java -javaagent:opentelemetry-javaagent.jar \
-Dotel.service.name=orders \
-Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
-Dotel.traces.sampler=parentbased_traceidratio -Dotel.traces.sampler.arg=0.1 \
-jar orders.jar
# HTTP servers/clients, JDBC, Kafka and Redis calls become spans; traceparent is propagated automatically.Span span = tracer.spanBuilder("fraud.check").setAttribute("order.id", orderId).startSpan();
try (Scope ignored = span.makeCurrent()) {
return fraudClient.check(order);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}Trade-offs
| Benefits | Costs | |
|---|---|---|
| Insight | Shows causality across services — which call was slow, which failed. | Only as complete as propagation: one uninstrumented hop breaks the trace. |
| Cost | Sampling keeps volume manageable. | Storage and collector capacity grow with traffic and span count. |
| Effort | Auto-instrumentation covers common libraries. | Async work, queues and custom protocols need manual context propagation. |
Production considerations
- Propagate context through message headers too, so producer and consumer spans join one trace.
- Put the trace id in every log line and in error responses (centralized logging).
- Link traces from latency and error dashboards (exemplars) so an alert leads straight to example requests.
- Avoid putting personal data in span attributes.