Observability that actually helps during incidents
Signals worth paying for, dashboards worth keeping, and the questions your telemetry should answer in the first five minutes.
Most observability stacks can answer any question given enough time. During an incident you do not have time, so the only signals that matter are the ones that answer a fixed set of questions immediately.
The first five minutes
- Is this us or upstream?
- Which surface is affected, and how badly?
- What changed in the last hour?
Every dashboard that does not help answer one of these should not be on the incident path.
Cardinality is a budget, not a limit
High cardinality labels are useful exactly once: when they let you split a symptom by the dimension that explains it. Keep tenant, route, and version. Drop request ids and raw URLs from metrics and put them in traces and logs instead.
Alerts follow user impact
Alert on symptoms your users would describe, then use dashboards for causes. A single symptom alert with a good runbook beats twelve cause alerts nobody trusts.
Topics