Microservice Patterns
Metrics and Monitoring
Measure every service with a few standard signals — RED for requests, USE for resources — and alert on symptoms users feel, not on every cause you can think of.
What you will learn
- Apply the RED and USE methods to choose metrics.
- Explain why percentiles beat averages for latency.
- Alert on user-facing symptoms and use resource metrics for diagnosis.
- Define SLOs and alert on error-budget burn.
Overview
A car dashboard does not show every sensor: speed, fuel, temperature and a few warning lights. Drivers react to "engine overheating", not to the coolant pump's RPM. Good monitoring works the same way: a few clear signals, and alarms for what actually matters.
Metrics are numbers aggregated over time — cheap to store, ideal for trends and alerts. Two methods tell you which to collect. RED for every service: Rate (requests/s), Errors (failed/s), Duration (latency percentiles). USE for every resource (CPU, pool, queue, disk): Utilisation, Saturation (queued work), Errors.
Run it
Tiles turn red at typical alert thresholds. In A traffic grows past capacity: watch which signals change first. In B the payment provider fails: compare the error tile with utilisation.
What to alert on
| Signal | Use for | Alert? |
|---|---|---|
| Error rate, p95/p99 latency (symptoms) | "Are users hurting?" | Yes — page a human. |
| SLO error-budget burn rate | "Will we miss the objective?" | Yes — fast burn pages, slow burn tickets. |
| Saturation (queue length, pool wait) | Early warning before latency rises | Warn / ticket. |
| CPU, memory, GC (causes) | Diagnosis and capacity planning | Rarely — they fire without user impact and miss many real problems (scenario B). |
Instrumentation and alert rules
management.endpoints.web.exposure.include=health,prometheus
management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.slo.http.server.requests=100ms,300ms,1s
management.metrics.tags.service=ordersgroups:
- name: orders-slo
rules:
- alert: OrdersHighErrorRate
expr: sum(rate(http_server_requests_seconds_count{service="orders",status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_count{service="orders"}[5m])) > 0.02
for: 5m
labels: { severity: page }
- alert: OrdersSlow
expr: histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{service="orders"}[5m]))) > 0.3
for: 10m
labels: { severity: page }Trade-offs
| Benefits | Costs | |
|---|---|---|
| Cost | Metrics are cheap compared with logs and traces. | High-cardinality labels (user id, URL with ids) explode storage. |
| Alerting | Symptom alerts are few and always matter. | They fire when users are already affected — add saturation warnings for lead time. |
| Detail | Perfect for trends, capacity and SLOs. | Say what is wrong, not why — follow up with traces and logs. |
Production considerations
- One standard RED dashboard per service, generated from the same template.
- Every paging alert needs a runbook link and must be actionable; delete alerts nobody acts on.
- Keep label cardinality bounded: route templates (
/orders/{id}), not raw paths. - Use exemplars to jump from a latency spike to example traces.