Instrumenting with OpenTelemetry: From Collection to Alerting in Jujing OBSERVE

OpenTelemetry (OTel) has become the de facto standard for instrumentation and telemetry collection in cloud-native environments. Its real value isn't a new protocol—it's that it collapses logs, metrics, and traces into a single data model, so you stop rewriting your collection code every time you switch backends. This article uses Java and Python as representative stacks and walks through wiring data into Jujing OBSERVE and standing up your first alert.

Deploy the OpenTelemetry Collector

The Collector receives telemetry from your applications, batches it, and forwards it onward. For a quick local test you can run the binary directly; in production, run it in containers or under systemd. Here is a minimal config.yaml that forwards OTLP data to OBSERVE:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
exporters:
  otlphttp:
    endpoint: https://ob.jjhub.cn/otel/v1/traces
    headers:
      Authorization: "Bearer <your-token>"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Two things are worth calling out. First, export endpoints are split by signal type—traces and metrics use different paths, so mirror that in your exporter configuration. Second, keep the Collector in the same network region as your applications in production; crossing the public internet introduces latency and packet loss you will have to budget for.

Instrument your applications

For Java, the opentelemetry-javaagent collects telemetry without touching business code:

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.traces.exporter=otlp \
  -Dotel.metrics.exporter=otlp \
  -Dotel.logs.exporter=otlp \
  -Dotel.exporter.otlp.endpoint=http://collector:4317 \
  -jar app.jar

For Python, opentelemetry-instrument injects automatically across Flask, FastAPI, requests, psycopg2, and other common libraries:

opentelemetry-instrument \
  --traces_exporter otlp \
  --metrics_exporter otlp \
  --exporter_otlp_endpoint http://collector:4317 \
  gunicorn app:app

Think about sampling before you ship. Auto-instrumentation captures every request by default, which is fine in staging but expensive at scale—on a high-traffic path it can overwhelm the Collector. Once you have a baseline, switch to a parent-based or tail-based sampler so the noisy paths don't drown everything else.

One gotcha worth calling out early: set the service.name resource attribute explicitly rather than relying on the default. Auto-instrumentation derives it from the process or entrypoint, and that default is rarely consistent across environments, which makes traces hard to search and alerts hard to attribute later. In Java, set -Dotel.resource.attributes=service.name=order-service; in Python, export OTEL_RESOURCE_ATTRIBUTES=service.name=order-service. Consistent service names are the difference between a trace view that groups correctly and one that scatters spans across pseudo-services.

Correlate trace_id into your logs

The gap between traces and logs is where most troubleshooting time disappears. The fix is to make your logging framework carry trace_id and span_id. With Logback, for example:

<pattern>%d{HH:mm:ss.SSS} [%X{trace_id:-}] [%X{span_id:-}] %-5level %logger{36} - %msg%n</pattern>

This works because OTel's Logback MDC instrumentation (otel.instrumentation.logback-mdc) injects the current span context into the MDC at runtime. Once enabled, you can open a trace in OBSERVE and jump straight to its associated log lines—no more grepping by timestamp and hoping the clock skew between hosts is small.

Ship your first alert

Resist the urge to configure a wall of rules the moment data starts flowing. Start with two high-value metrics: service error rate (5xx ratio above 2% for 3 minutes) and P99 latency (above 500ms for 5 minutes). Route alerts to WeCom Work or DingTalk so nobody is stuck reading email at 2 a.m., and add a cooldown so a single flapping service doesn't page the whole on-call roster.

The error-rate rule in practice looks like sum(rate(http_requests_total{status=~"5.."}[3m])) / sum(rate(http_requests_total[3m])) > 0.02, with a three-minute for clause so a transient blip doesn't fire. For latency, the equivalent is histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5. Wire both to the same escalation path and you have a sane first pass that covers availability and tail latency without drowning on-call.

A quick sanity check once data is flowing: open the trace explorer and confirm you can see a recent request end-to-end, then open its logs from the trace view. If that round trip works, your pipeline is healthy. Finally, run instrumentation in staging for at least a week before going live—confirm your sampling rate, metric baselines, and Collector resource usage all look sane before rolling out to production. If traces appear but metrics do not, re-check the exporter endpoint paths first—it is the most common silent failure.