Distributed Tracing Explained: From a Slow Request to Root Cause

Why logs and metrics can't find a slow request

An order endpoint's P99 latency jumps from 300ms to 2s in production. Every log line shows a normal return; every metric just says "slower" without saying where. One request passes through a gateway, order, inventory, and payment service, then lands on a database and a cache — any one of those hops could be the culprit. Tracing's idea is simple: assign each request a globally unique trace_id, stitch the cross-service calls into a single chain, and the longest hop and the failing hop become visible the moment you expand it.

Three concepts: trace, span, and context propagation

• Trace: the full chain of one request, identified by a trace_id.
• Span: one unit in that chain — a single HTTP call or a single SQL statement — recording start, end, duration, and tags.
• Context propagation: passing trace_id and span_id downstream through request headers so spans scattered across services can be reassembled into one trace.

OBSERVE's backend follows the OpenTelemetry standard, so any OTel-conforming data lands directly without tying you to a specific SDK. You instrument your app and hand the data to OBSERVE — no format conversion in between. That's also why you'd pick it: if you switch backends later, the instrumentation code doesn't change.

Four steps to locate a slow request

A trace expands in OBSERVE into a call tree, and locating a slow request follows a fixed path:

1. Retrieve the target trace by trace_id, or by service plus a time range.
2. Look at the per-span duration distribution in the tree and circle the longest one.
3. Open the span and see which downstream calls it made and what tags and events it carries.
4. Correlate logs and metrics within that span's time window to confirm whether it's a slow DB, network jitter, or code.

Steps three and four are where the payoff is. The span carries the SQL and its duration, and right beside it is the slow-query log for that exact statement — no flipping between the logging and tracing systems. A slow SQL is the most common answer to "where is it slow", and this correlation takes you from "the endpoint is slow" to "this query is slow" in two clicks.

A worked example

Take a concrete case: a payment-path trace whose call tree shows a gateway span at 12ms, an order-service span at 40ms, and a payment.charge span at 1.9s. Inside that span, one child — INSERT INTO ledger ... — took 1.85s. That's the whole story on one screen: gateway and order service are healthy, payment is fine, and a single ledger insert (likely a missing index or a lock wait) ate the time. Without tracing you'd spend the afternoon bisecting which of five services to blame.

Service topology and dependency analysis

Aggregate all traces and you get a service dependency map for free: who calls whom, at what volume, error rate, and average latency. It's invaluable for understanding a system at a glance and for inheriting an undocumented legacy stack. When a node turns red on the topology, its error rate or latency has gone wrong; click through and you get the trace samples behind it.

The map also exposes hidden dependencies: an old service that should have been decommissioned long ago but is still receiving calls shows up plain as day. These findings are hard to surface by reading code and trivial to see on the map.

Sampling: balancing cost and coverage

Full tracing is expensive; fixed-ratio sampling risks missing anomalies. OBSERVE recommends tail sampling: ordinary requests are sampled at a fixed ratio (say 10%), while slow and failing requests are kept at 100%. Cost stays controlled and every "worth looking at" request keeps a complete trace. Sampling policies can be set per service — higher on critical paths, lower on edge services. The rule of thumb: sample enough that no slow or failed request ever goes unsampled, and let the healthy majority carry the cost reduction.

Practical recommendations

• Start instrumentation from the core path: trace order placement and payment first, then roll out gradually.
• Standardize span naming across teams — otherwise your topology and search both degrade into noise.
• Put the trace_id into your application logs. That's what actually links logs and traces, so a single trace_id in one log line lets you reconstruct the whole chain during an incident.