Microservice Patterns
Centralized Logging
Ship every service's logs to one searchable place as structured events that carry a correlation id, so one failed request can be followed across all the services and instances it touched.
What you will learn
- Explain why per-host log files fail in distributed systems.
- Write structured logs with correlation (trace) ids.
- Follow one failed request across services and instances.
- Control log volume, cost and sensitive data.
Overview
If every shop in a chain keeps its own paper receipt book, finding out what happened to one customer's order means visiting every shop. Put all receipts in one searchable system with the order number on each, and you type the number once.
In a distributed system one user action produces log lines on many services and instances, which may be restarted or replaced at any time. Centralized logging ships every line to one store (Elasticsearch/OpenSearch, Loki, a cloud service). Two habits make it useful: structured logs (JSON fields, not free text) and a correlation id — ideally the trace id — on every line.
Run it
Open the Trace tab and expand *Log lines with this trace id*. In A, find which service is behind the gateway's generic errors. In B, three instances of orders look the same in aggregate — the log lines of failed requests tell them apart.
Fields every log line should have
| Field | Why |
|---|---|
timestamp (UTC, ms) | Ordering events across hosts. |
service, instance, version | Which deployment wrote it — bad instance or release (scenario B). |
trace_id, span_id | Join all lines of one request; jump to its trace. |
level, message, event | Severity and a stable, searchable event name. |
Business ids (order_id) | Answer support questions — never secrets or full personal data. |
Structured logs with the trace id
logging.structured.format.console=ecs
logging.structured.ecs.service.name=orders
logging.structured.ecs.service.version=${APP_VERSION}
management.tracing.sampling.probability=0.1
# Every log line now carries trace.id and span.id — search: trace.id:"4bf92f3577b34da6"log.atWarn()
.setMessage("payment declined")
.addKeyValue("event", "payment.declined")
.addKeyValue("order_id", order.id())
.addKeyValue("provider_code", response.code())
.log();
// Return the trace id to the client in error responses so support can find the logs.Related operational logs
- Audit logging: record who did what to which business object and when (refunds, permission changes, data exports) in an append-only store with longer retention — separate from debug logs, and queryable for compliance.
- Exception tracking: send unhandled exceptions to a tracker (Sentry, Rollbar, App Insights) that groups duplicates, counts affected users and alerts on new error types — search logs for detail, the tracker for trends.
- Log deployments and changes: emit an event for every deploy, config change and feature-flag flip, and show them as markers on dashboards; most incidents start right after a change.
Trade-offs
| Benefits | Costs | |
|---|---|---|
| Debugging | One search shows a request's full story across services. | Only if every service logs the correlation id consistently. |
| Cost | Retention tiers keep old logs cheap. | Ingest and indexing cost grows with volume; verbose logs get expensive. |
| Signal | Rich detail for individual events. | Poor for trends — use metrics for rates and percentiles. |
Production considerations
- Write logs to stdout and let the platform ship them; never depend on files on the instance.
- Agree on field names across services (a shared logging config or ECS / OpenTelemetry semantic conventions).
- Sample or rate-limit noisy logs instead of disabling them.
- Alert on metrics, investigate with logs and traces.