Skip to content
System Design Lab
Microservice Patterns

Metrics and Monitoring

Measure every service with a few standard signals — RED for requests, USE for resources — and alert on symptoms users feel, not on every cause you can think of.

Intermediate 4 min read· Updated 2026-11-18

What you will learn

  • Apply the RED and USE methods to choose metrics.
  • Explain why percentiles beat averages for latency.
  • Alert on user-facing symptoms and use resource metrics for diagnosis.
  • Define SLOs and alert on error-budget burn.

Overview

A car dashboard does not show every sensor: speed, fuel, temperature and a few warning lights. Drivers react to "engine overheating", not to the coolant pump's RPM. Good monitoring works the same way: a few clear signals, and alarms for what actually matters.

Metrics are numbers aggregated over time — cheap to store, ideal for trends and alerts. Two methods tell you which to collect. RED for every service: Rate (requests/s), Errors (failed/s), Duration (latency percentiles). USE for every resource (CPU, pool, queue, disk): Utilisation, Saturation (queued work), Errors.

Run it

Tiles turn red at typical alert thresholds. In A traffic grows past capacity: watch which signals change first. In B the payment provider fails: compare the error tile with utilisation.

Loading simulation…

What to alert on

SignalUse forAlert?
Error rate, p95/p99 latency (symptoms)"Are users hurting?"Yes — page a human.
SLO error-budget burn rate"Will we miss the objective?"Yes — fast burn pages, slow burn tickets.
Saturation (queue length, pool wait)Early warning before latency risesWarn / ticket.
CPU, memory, GC (causes)Diagnosis and capacity planningRarely — they fire without user impact and miss many real problems (scenario B).

Instrumentation and alert rules

Spring Boot + Micrometer: RED metrics with histograms
management.endpoints.web.exposure.include=health,prometheus
management.metrics.distribution.percentiles-histogram.http.server.requests=true
management.metrics.distribution.slo.http.server.requests=100ms,300ms,1s
management.metrics.tags.service=orders
Prometheus: alert on symptoms
groups:
  - name: orders-slo
    rules:
      - alert: OrdersHighErrorRate
        expr: sum(rate(http_server_requests_seconds_count{service="orders",status=~"5.."}[5m]))
              / sum(rate(http_server_requests_seconds_count{service="orders"}[5m])) > 0.02
        for: 5m
        labels: { severity: page }
      - alert: OrdersSlow
        expr: histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{service="orders"}[5m]))) > 0.3
        for: 10m
        labels: { severity: page }

Trade-offs

BenefitsCosts
CostMetrics are cheap compared with logs and traces.High-cardinality labels (user id, URL with ids) explode storage.
AlertingSymptom alerts are few and always matter.They fire when users are already affected — add saturation warnings for lead time.
DetailPerfect for trends, capacity and SLOs.Say what is wrong, not why — follow up with traces and logs.

Production considerations

  • One standard RED dashboard per service, generated from the same template.
  • Every paging alert needs a runbook link and must be actionable; delete alerts nobody acts on.
  • Keep label cardinality bounded: route templates (/orders/{id}), not raw paths.
  • Use exemplars to jump from a latency spike to example traces.

Further reading

Related topics