Skip to content
System Design Lab
Microservice Patterns

Distributed Tracing

Follow one request through every service it touches as a tree of timed spans, so you can see exactly which hop made it slow or failed — something averages and per-service logs cannot show.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Explain traces, spans and context propagation.
  • Read a trace waterfall: nesting, sequential vs parallel calls, gaps.
  • Find a slow dependency and an N+1 query from traces.
  • Instrument services with OpenTelemetry and choose a sampling strategy.

Before you start

Overview

A parcel tracking page shows every stop a parcel made and how long it waited at each depot. If it arrived late, you can see that it sat in one warehouse for two days. A trace is the same for a request: every service it passed through, and how long each took.

A trace is one request's journey. Each unit of work — handling an HTTP request, a database query, a call to another service — is a span with a start, a duration, a status and a parent. Services pass the trace id and parent span id along with every call (context propagation, standardised as the W3C traceparent header), so a tracing backend can rebuild the tree.

Run it

Open the Trace tab while the simulation runs. A shows a healthy checkout. In B, p99 is high but nothing errors — pick *Slowest recent* to find the culprit. C hides an N+1 query.

Loading simulation…

Reading a waterfall

  • Width is duration; the widest leaf span is usually the cause, its ancestors are only waiting.
  • Staircases of many short spans to the same component mean a loop of calls (N+1 queries).
  • Spans one after another that could run in parallel are free latency savings.
  • Gaps between a parent's start and its children are time spent inside the parent (CPU, locks, queueing).

Instrumentation with OpenTelemetry

Zero-code instrumentation for a Spring Boot service
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=orders \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -Dotel.traces.sampler=parentbased_traceidratio -Dotel.traces.sampler.arg=0.1 \
  -jar orders.jar
# HTTP servers/clients, JDBC, Kafka and Redis calls become spans; traceparent is propagated automatically.
A custom span around business logic
Span span = tracer.spanBuilder("fraud.check").setAttribute("order.id", orderId).startSpan();
try (Scope ignored = span.makeCurrent()) {
    return fraudClient.check(order);
} catch (Exception e) {
    span.recordException(e);
    span.setStatus(StatusCode.ERROR);
    throw e;
} finally {
    span.end();
}

Trade-offs

BenefitsCosts
InsightShows causality across services — which call was slow, which failed.Only as complete as propagation: one uninstrumented hop breaks the trace.
CostSampling keeps volume manageable.Storage and collector capacity grow with traffic and span count.
EffortAuto-instrumentation covers common libraries.Async work, queues and custom protocols need manual context propagation.

Production considerations

  • Propagate context through message headers too, so producer and consumer spans join one trace.
  • Put the trace id in every log line and in error responses (centralized logging).
  • Link traces from latency and error dashboards (exemplars) so an alert leads straight to example requests.
  • Avoid putting personal data in span attributes.

Further reading

Related topics