Debugging latency: a practical playbook
A narrowing procedure that moves from user-visible latency to a specific queue, lock, or dependency.
Latency debugging is a narrowing exercise. Each step should cut the search space in half, and each step should produce evidence you can paste into a thread.
Narrow by surface, then by hop
Start with the slowest user-visible surface, then walk the request path one hop at a time. Compare p50 and p99 at every hop: a p99-only regression points at queuing or contention, while a shifted p50 points at work that got genuinely more expensive.
Look for the queue
Most latency lives in a queue: a connection pool, a thread pool, a disk, or a lock. Find the resource whose utilization tracks the latency curve and you have found the queue.
Confirm with a change
The last step is always a controlled change: raise a pool size, remove a lock, or shed load, and confirm the curve moves. Without that step you have a hypothesis, not a diagnosis.
Topics