Correlating Logs and Traces with trace_id: An OpenTelemetry Walkthrough

Debugging a timeout usually takes three steps: spot the slow request, follow the trace to the specific span, then go to the logs to find out what that span actually errored on. Without a trace_id in the logs, step three is a needle in a haystack. This article shows how to wire logs and traces together with OpenTelemetry.

The core of correlation: put trace_id in every log line

The idea is simple: when the application produces a trace, inject the current trace_id and span_id into the logging context, and let the logging library emit both fields automatically. With Python's opentelemetry-instrumentation-logging:

import logging
from opentelemetry.instrumentation.logging import LoggingInstrumentor

LoggingInstrumentor().instrument(set_logging_format=True)
logger = logging.getLogger(__name__)

After this, log output automatically carries otelTraceID and otelSpanID. Java uses opentelemetry-logback-appender, attached in logback.xml; Go just adds two fields to the logrus or zap formatter. The common thread is that none of them require touching business code — one configuration change at the logging framework layer is enough.

The important part is not the instrumentation itself — it is emitting these two values as structured fields instead of concatenating them into the message string. Concatenated values force you to search with regex, which is expensive and fragile; structured fields can be indexed and filtered exactly.

Let logs form part of the trace

OpenTelemetry's Logs signal, like Metrics, can carry trace_id and span_id attributes. Configure the OTLP exporter to ship logs and traces to the same backend, and the backend can attach logs to the corresponding span:

exporter:
  otlp:
    endpoint: "ob.example.com:4317"
    protocol: grpc

Jujing OBSERVE indexes by trace_id and lists every log a request produced directly on the trace detail page. Open a slow span and its ERROR lines sit right below it — no switching to the log page and guessing keywords. The search API also supports exact filtering by trace_id, so one query pulls all logs for a request.

Verifying it works is quick: fire a request that logs something, grab its trace_id, then filter logs by that trace_id to see if those lines come back; next, open the matching span on the trace detail page and confirm the log panel has content. Both ends have to work before correlation counts as done.

The usual traps: async tasks and cross-thread context loss

Async code is where log correlation breaks most often. In thread pools, message queues, and scheduled tasks the context has already switched, the trace_id goes with it, and logs no longer correlate.

Two fixes: first, pass context explicitly with OpenTelemetry's Context.attach() to carry the parent span's context into the new thread; second, put the trace_id in the queue message manually and restore it with set_span_in_context on the consumer side. Do not expect thread-local variables to hop threads for you — they won't.

The other trap is the gateway. If a request passes through Nginx, Kong, or an API gateway, forward the upstream trace_id downstream — otherwise the gateway's span and the application's span become two separate trees. The header to forward is the standard W3C traceparent; make sure the gateway does not drop it like an ordinary header.

Sampling matters too. If you only keep head-based sampling, the traces that errored may be exactly the ones sampled away, and logs cannot attach to the critical scene. Add tail-based sampling for error and high-latency requests so the full trace plus logs survive for slow and failing requests.

Troubleshooting: trace_id still missing from logs

If the fields still don't show up after the setup above, check these in order: first, whether instrumentation actually loaded — look for OpenTelemetry initialization lines in the startup log; second, the log format — confirm the formatter includes the otelTraceID field; third, the exporter — make sure both the logs and traces pipelines are configured, since sending traces without logs is a common omission; fourth, sampling — if head sampling dropped this trace, the logs have nothing to attach to. Going through the list usually finds it.

Suggestions for rollout

• Standardize on the OTel Logs signal instead of each language inventing its own format.
• Make trace_id a structured field and build the index.
• Pass context explicitly across async boundaries and carry trace_id in queue messages.
• Prove it on one core service first, then roll it out everywhere.

Once logs and traces are linked, an incident goes from "guess" to "two clicks and you see it." That is a better investment than adding more machines.