← Back to blog
Product 4 min read 炬鲸团队

Unified Logs, Metrics, and Traces: Tracing a Slow Query End to End

A 2 a.m. slow-query alert walks through how OBSERVE links logs, metrics, and traces on trace_id, narrowing the root cause in three steps and cutting MTTR from hours to minutes.

Why Put Three Data Types in One Place

Classic troubleshooting runs in three separate passes: check CPU, memory, and QPS in the monitoring tool, search keywords in the log platform, then finally capture a trace by hand. The three systems don't share a timeline, field names don't match, and a single production incident means switching between three consoles just to line up timestamps. In our own on-call reviews, operators routinely spent the first ten to fifteen minutes of an incident aligning clocks across tools before they could even start reasoning about the root cause.

OBSERVE takes the opposite approach. Logs, metrics, and traces go into one data model and are joined on common fields like trace_id, span_id, service, and host. The payoff is immediate: from any one screen you can jump to the other two data types without guessing which request a given log line belongs to.

This correlation is more than slapping three pages behind one menu. Two things have to happen underneath. First, the collectors must inject trace_id into log lines and metric labels consistently. Second, the query engine must join across data sources on those shared fields so that a cross-source query stays correct within a single time window. Get either one wrong and the "correlation" is just a demo that falls apart the first time someone actually needs it in the middle of an incident.

Starting from a Slow-Query Alert

Say you get paged at 2 a.m.: the average latency of SELECT * FROM orders WHERE status = ? on the order database jumped from 120ms to 3.2s, and P99 is worse at 8s. The old way is to connect to the database and read the slow-query log, then dig through app logs, then remember to look at traces. Our way is to click straight into the alert card.

The alert carries a sample trace_id, which opens a trace view: gateway → order service → order DAO → MySQL. The flame graph shows 97% of the time spent in MySQL's ExecQuery, so the problem is not in the application layer. That single click already saved the detour through the app logs, and it told you exactly which hop to focus on next.

How Logs, Metrics, and Traces Connect

The trace tells you the database is slow. The next question is why. On the span detail page, click "Related logs" and the system pulls the order service's logs for that trace_id window. One entry is a slow-query log with the full SQL and the execution plan it actually hit — no guessing, no manual grep.

Then switch to metrics. Over the same window, the MySQL instance shows flat QPS, but Innodb_row_lock_waits climbed from 0 to more than 200 per second. Put the three signals together and the picture is clear: the query did not get slower; last night's release changed the transaction isolation level, and row-lock waits are dragging the SQL down. Rolling back the config restores latency. Notice how each data type answered a different part of the question — the trace found the slow span, the log found the exact SQL, and the metric found the lock contention.

What Makes the Correlation Actually Work

The mechanics matter more than the UI. Under the hood, every span is indexed by trace_id and span_id, every log line carries those same two fields plus a timestamp, and every metric series is tagged with service and host. When you click "Related logs" on a span, the platform is not running a fuzzy full-text search — it is doing a point lookup on trace_id within the span's time window. That is why it returns the exact lines instead of a wall of loosely matching results, and why it stays fast even on busy services. If your team adopts the platform without enforcing these fields in the collectors, the correlation silently degrades into exactly the three-console experience you were trying to escape.

Practical Notes

  1. Propagate trace_id end to end — gateway, services, and database middleware. One missing hop breaks the chain.
  2. Always emit trace_id and span_id in logs, or "correlation" is just a checkbox.
  3. Label metrics by service and host so dimensions line up with traces and logs.
  4. Don't set a uniform sampling rate. Sample the critical path heavily so correlation doesn't lose its samples.

None of this is glamorous work. It is the plumbing that decides whether an on-call engineer gets an answer in minutes or an exhausting afternoon. Whether your MTTR drops from hours to minutes often comes down to whether your data actually connects — the difference between three open consoles and one answer.