Skip to content
System Design Lab
Microservice Patterns

Centralized Logging

Ship every service's logs to one searchable place as structured events that carry a correlation id, so one failed request can be followed across all the services and instances it touched.

Beginner 4 min read· Updated 2026-11-18

What you will learn

  • Explain why per-host log files fail in distributed systems.
  • Write structured logs with correlation (trace) ids.
  • Follow one failed request across services and instances.
  • Control log volume, cost and sensitive data.

Overview

If every shop in a chain keeps its own paper receipt book, finding out what happened to one customer's order means visiting every shop. Put all receipts in one searchable system with the order number on each, and you type the number once.

In a distributed system one user action produces log lines on many services and instances, which may be restarted or replaced at any time. Centralized logging ships every line to one store (Elasticsearch/OpenSearch, Loki, a cloud service). Two habits make it useful: structured logs (JSON fields, not free text) and a correlation id — ideally the trace id — on every line.

Run it

Open the Trace tab and expand *Log lines with this trace id*. In A, find which service is behind the gateway's generic errors. In B, three instances of orders look the same in aggregate — the log lines of failed requests tell them apart.

Loading simulation…

Fields every log line should have

FieldWhy
timestamp (UTC, ms)Ordering events across hosts.
service, instance, versionWhich deployment wrote it — bad instance or release (scenario B).
trace_id, span_idJoin all lines of one request; jump to its trace.
level, message, eventSeverity and a stable, searchable event name.
Business ids (order_id)Answer support questions — never secrets or full personal data.

Structured logs with the trace id

Spring Boot: JSON logs; Micrometer Tracing puts traceId/spanId in the MDC
logging.structured.format.console=ecs
logging.structured.ecs.service.name=orders
logging.structured.ecs.service.version=${APP_VERSION}
management.tracing.sampling.probability=0.1
# Every log line now carries trace.id and span.id — search: trace.id:"4bf92f3577b34da6"
Log events, not prose
log.atWarn()
   .setMessage("payment declined")
   .addKeyValue("event", "payment.declined")
   .addKeyValue("order_id", order.id())
   .addKeyValue("provider_code", response.code())
   .log();
// Return the trace id to the client in error responses so support can find the logs.
  • Audit logging: record who did what to which business object and when (refunds, permission changes, data exports) in an append-only store with longer retention — separate from debug logs, and queryable for compliance.
  • Exception tracking: send unhandled exceptions to a tracker (Sentry, Rollbar, App Insights) that groups duplicates, counts affected users and alerts on new error types — search logs for detail, the tracker for trends.
  • Log deployments and changes: emit an event for every deploy, config change and feature-flag flip, and show them as markers on dashboards; most incidents start right after a change.

Trade-offs

BenefitsCosts
DebuggingOne search shows a request's full story across services.Only if every service logs the correlation id consistently.
CostRetention tiers keep old logs cheap.Ingest and indexing cost grows with volume; verbose logs get expensive.
SignalRich detail for individual events.Poor for trends — use metrics for rates and percentiles.

Production considerations

  • Write logs to stdout and let the platform ship them; never depend on files on the instance.
  • Agree on field names across services (a shared logging config or ECS / OpenTelemetry semantic conventions).
  • Sample or rate-limit noisy logs instead of disabling them.
  • Alert on metrics, investigate with logs and traces.

Further reading

Related topics