A hands-on guide to adopting OpenTelemetry: why OTel over vendor SDKs, zero-code auto-instrumentation for Java and Node, custom spans for business logic, trace_id in logs, and a troubleshooting checklist.
The first decision when instrumenting is "whose SDK do I use". A self-built or vendor-locked SDK means rewriting your instrumentation the day you switch backends. OpenTelemetry (OTel) is a CNCF project and a single API: instrument once, and the same data can go to OBSERVE, Jaeger, and Prometheus simultaneously. OBSERVE speaks the OTel protocol (OTLP) natively, which makes this the cheapest path in and the hardest to get locked into.
Java, Node, Python, and Go all have OTel auto-instrumentation agents that work via bytecode injection or monkey-patching, creating spans automatically at framework choke points — HTTP entry, DB drivers, message-queue clients — with no application code changes.
For Java, add a javaagent to the launch command:
java -javaagent:opentelemetry-javaagent.jar \
-Dotel.exporter.otlp.endpoint=https://ob.jjhub.cn/ingest/otlp \
-Dotel.service.name=order-service \
-jar order-service.jarOnce it starts, HTTP requests, JDBC queries, and Redis calls produce spans and report automatically. Auto-instrumentation covers roughly 80% of common cases; the remaining 20% is time spent inside business logic, which needs custom spans.
With just the agent on the classpath, you typically get spans for inbound and outbound HTTP, JDBC and Redis operations, and Kafka/AMQP message production and consumption — each with the method, the endpoint or topic, and timing. That's most of what you need to answer "which downstream is slow", before you write a single line of instrumentation code. The naming follows the framework, so the span names are consistent across every service that uses the same stack.
Auto-instrumentation can't see inside a business function — "this loop computing coupons took 800ms" is invisible to it. That's where manual spans come in:
Span span = tracer.spanBuilder("calc-coupon")
.setAttribute("coupon.count", 5)
.startSpan();
try (Scope scope = span.makeCurrent()) {
calculateCoupons(order);
} finally {
span.end();
}Name spans meaningfully so they answer "what is this operation" — never span1, span2. When adding attributes, keep sensitive data out: phone numbers and ID numbers get stored and indexed, so putting them on a span is equivalent to writing them into your tracing store. If a value is sensitive enough to mask in logs, it's sensitive enough to leave off a span.
The most practical way to link tracing and logs is to carry the trace_id in log lines. Use OTel's log bridge, or add %X{trace_id} to your Logback pattern via MDC:
<pattern>%d %-5level [%thread] %X{trace_id} %logger{36} - %msg%n</pattern>Then a single error log in OBSERVE gives you the trace_id to reconstruct the whole chain — the direction you'll actually follow most often during an incident, since you usually find the failing log first, not the trace.
Auto-instrumentation costs roughly 1-5% CPU per request, plus a small packet per span on the network. For production: 100% sampling on core services (paired with tail sampling), 10-30% on edge services, and let the agent batch and report asynchronously so business threads never block on export.
Don't flip the whole fleet to 100% in one day. Start with one service, watch its CPU and the ingest volume for a week, then expand. The agent's default batching is usually fine, but if you see export latency climbing, raise OTEL_BSP_MAX_QUEUE_SIZE before touching the sampling ratio — the bottleneck is usually the exporter queue, not the sampler.
If you deploy on Node or Python, the mechanism differs but the pattern is the same. Node loads the tracing module with a --require flag at startup; Python wraps your entry point with opentelemetry-instrument. The key environment variables — OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME — are identical across languages, so one runbook works whether the service is Java, Node, or Go. Keep those two variables in a single configmap or secret and every service inherits the correct endpoint without per-language drift.