Onboarding OpenTelemetry to OBSERVE: Collector, SDK, and Log Correlation

Why OpenTelemetry

OpenTelemetry (OTel) is the mainstream standard for collecting observability data today: one SDK emits traces, metrics, and logs, so you don't have to wire up a different vendor's instrumentation library in your code. OBSERVE is fully compatible with the OTLP protocol, so onboarding is just "add a dependency and change some config" — no business-logic changes. It also means the instrumentation is portable: if you ever switch backends, the code keeps emitting OTel and you only change the exporter destination.

Step 1: Deploy the Collector

The Collector is the recommended first stop for all telemetry. It decouples your apps from the backend, lets you batch and filter data before it leaves the network, and gives you one place to inject credentials. In production, run one per host or Kubernetes node. The core config looks like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
exporters:
  otlphttp:
    endpoint: https://ob.jjhub.cn/otlp
    headers:
      X-Tenant: platform
      Authorization: Bearer <token>
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

The batch processor merges scattered spans into batches before sending, which noticeably cuts network overhead and export pressure. For logs you would add an otlp log pipeline the same way.

Step 2: Add the SDK to Your App

For Java apps, the opentelemetry-javaagent is usually the way — one extra startup flag and zero code changes:

java -javaagent:opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint=http://localhost:4317 -Dotel.service.name=order-service -jar app.jar

The agent auto-instruments common components like HTTP, databases, and message queues. Python, Go, and Node.js use their respective SDKs, with the exporter pointed at the Collector's 4317 port. Make sure service.name matches what you registered in the backend, or traces will land under an unexpected service name and get lost in the noise.

Step 3: Correlate Traces and Logs

Traces alone aren't enough — during troubleshooting you also want to link logs to spans. Make logs carry trace_id and span_id, and OBSERVE lets you jump from a single log line straight to its trace. For Logback:

<pattern>%d{yyyy-MM-dd HH:mm:ss} [%X{trace_id}/%X{span_id}] %-5level %logger{36} - %msg%n</pattern>

With OTel's automatic MDC injection, %X{trace_id} gets filled in for you — no manual wiring. Without this step you get two disconnected views; with it, an error in the log is one click from the slow span that caused it.

Running the Collector as a DaemonSet

On Kubernetes, the cleanest pattern is a DaemonSet — one Collector per node, apps send to localhost — rather than a central gateway for everything. It keeps egress local and survives a node restart alongside the pods. If you prefer a sidecar per pod, reuse the same config and point the app at localhost:4317 inside the pod. Either way, mount the config as a ConfigMap and pin the image tag deliberately: a rolling DaemonSet update is your main risk surface during upgrades, so pin a version instead of floating latest.

Sampling, Not All-or-Nothing

Collecting 100% of traces from every service is expensive and rarely useful. Use a head-based sampler so the sampling decision propagates across services, keeping whole traces intact rather than fragments. A common split is 100% for the core checkout or trading path, 10% for ordinary endpoints, and zero for health checks and static assets. For rare-but-critical requests you can force sampling with a per-request attribute, then find those traces back from a business key later.

Troubleshooting the First Run

If OBSERVE shows no data, check in order: is the Collector actually listening on 4317 (ss -lntp | grep 4317)? Does the app point at the Collector, not directly at OBSERVE? Is the auth token still valid in the exporter headers? A quick way to isolate the failure is to swap in the debug exporter temporarily — if spans print locally, the problem is downstream of the Collector, not in your app.

Practical Advice

Onboard traces first, then metrics, then unify log formats — spreading everything at once makes problems hard to isolate. Before going live, validate the data format locally with the Collector's debug exporter, then switch to the OBSERVE production endpoint. Keep an eye on the Collector's own otelcol_exporter_sent_spans metric to confirm data is actually flowing before you declare the integration done.