Instrument Your Service with OpenTelemetry in Ten Minutes

Goal and Prerequisites

The goal is concrete: take a Java or Python service running in a container and, within ten minutes, ship traces, metrics, and logs into OBSERVE with a full call chain visible in the UI. There's one prerequisite — OBSERVE is already deployed and you have the OTLP endpoint and access token. The rest assumes your service runs in a container and you can change its startup command or environment variables; zero business-code changes are required for the first step.

Step 1: Install the SDK and Enable Auto-Instrumentation

For Java you don't touch business code; just add a -javaagent flag:

java -javaagent:opentelemetry-javaagent.jar   -Dotel.exporter.otlp.endpoint=http://observe-host:4317   -Dotel.resource.attributes=service.name=order-api,deployment.environment=prod   -Dotel.metrics.exporter=otlp   -jar order-api.jar

For Python, install the distro and bootstrap the packages:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument   --traces_exporter otlp --metrics_exporter otlp   --exporter_otlp_endpoint http://observe-host:4317   python app.py

The key point: the export endpoint uses OTLP throughout (gRPC on 4317 or HTTP on 4318). OBSERVE accepts both traces and metrics over that single endpoint, so there's no need to maintain two separate exporter configs. Always set service.name explicitly — otherwise it falls back to the process or jar name and nothing lines up when you search. In production, add deployment.environment too so you can filter by environment.

Step 2: Add Manual Spans for Business Logic

Auto-instrumentation covers frameworks and libraries, but business semantics like placing an order, processing a payment, or issuing a refund need explicit spans:

from opentelemetry import trace
tracer = trace.get_tracer("order-api")

with tracer.start_as_current_span("create_order") as span:
    span.set_attribute("order.amount", 128.0)
    span.set_attribute("user.id", uid)
    # ...business logic

Tagging spans with business attributes is what lets you filter down to a specific request by amount or user ID during an investigation. The Java equivalent is Span.current().setAttribute(...). One piece of advice: standardize attribute keys across the team — agree on user.id and order.amount up front — otherwise one engineer writes userId and another writes user_id, and your searches silently miss data.

Step 3: Correlate Logs with Traces

Inject the trace_id into your log format so every log line can be joined back to a call chain:

import logging
from opentelemetry import trace

class TraceIdFilter(logging.Filter):
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, '032x')
        return True

logging.basicConfig(format='%(asctime)s [trace_id=%(trace_id)s] %(message)s')

If your logs are already collected into OBSERVE by an agent like Filebeat, having a trace_id= field in the line is enough for the platform to auto-correlate it with the matching trace — no extra mapping to configure.

Sampling: Don't Capture Everything Up Front

Full trace sampling is expensive. Start with head sampling that keeps errors and slow requests:

• Sample normal requests at 10%.
• Keep 100% of requests with HTTP status >= 500.
• Keep 100% of requests slower than 500ms.

That controls storage and collection cost while guaranteeing the important samples are there when something breaks. OBSERVE also supports tail-based sampling, which re-filters at the agent by error rate and slow-request ratio.

Resource Attributes: Don't Settle for Just a Service Name

Beyond service.name, fill in the deployment metadata so you stop asking "which environment, which machine?" during an investigation:

• service.version: the version number, so you can match it up when debugging a rollback.
• host.name / container.id: pinpoint the exact instance.
• deployment.environment: distinguish prod from staging.

These attributes ride along with every span and are directly filterable in search. Once they're in place, the question "which version of order-api is erroring?" is a single query instead of a dig through release notes. Standardize the attribute names across the team too — don't let everyone invent their own.

Optional: Add an OpenTelemetry Collector

Once you have many services, having every process dial OBSERVE directly gets hard to manage. Insert a Collector in between to batch, redact, and route:

receivers:
  otlp:
    protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: { endpoint: 0.0.0.0:4318 } }
processors:
  batch: {}
  attributes:
    actions:
      - key: user.phone
        action: delete
exporters:
  otlp:
    endpoint: observe-host:4317
    headers: { authorization: "Bearer ${TOKEN}" }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [attributes, batch], exporters: [otlp] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [otlp] }

The Collector gives you three things: a single place to attach the token, centralized field redaction, and batching to cut network overhead. On Kubernetes, deploy it as a DaemonSet.

Verification and Common Pitfalls

After wiring it up, send a few real requests and search the Trace page by service name to confirm the full call chain is visible; then check the metrics page to make sure http.server.duration and friends are populated. Three pitfalls come up repeatedly: a malformed exporter endpoint (missing path or wrong port), a network block (run telnet observe-host 4317 to confirm reachability), and a sampling rate too low to capture rare requests — temporarily switch to always_on while debugging. Finally, make sure the OTLP reports carry the token, or the gateway will silently reject them. Two more recurring issues: a Java Agent version mismatched with the business JDK fails silently — check the startup log for any OpenTelemetry output first; and HTTP/1.1 export backs up under heavy traffic, so prefer gRPC (4317) in production. If you see traces but an empty metrics page, the metrics exporter is likely off — check the otel.metrics.exporter setting.

Getting wired up is just the start — keep adding spans on critical paths, fill in business attributes, and bake it into CI.