Onboarding JJHub with OpenTelemetry: From Instrumentation to Traces

Getting traces, metrics, and logs into JJHub is a three-step job: instrument your app with the OpenTelemetry SDK, run a Collector to receive and forward the data, and let JJHub accept it over OTLP. This guide gives you the exact config and code to copy.

Overall architecture

Three steps: instrument your app with the OpenTelemetry SDK → run an OTel Collector in the cluster to receive, batch, and forward → JJHub accepts the OTLP protocol and stores it. We recommend the intermediate Collector instead of pointing apps directly at JJHub for a few reasons. It adds local buffering, redaction, and sampling, so data isn't lost when an app restarts and you don't have to maintain a push config inside every service. It also gives you one choke point for changing destinations, rotating tokens, or adding processors without touching application code.

Deploy it as a DaemonSet (one per node) if you're also scraping host metrics and log files; use a deployment behind a service if you only need the OTLP receiver. If you're migrating off a vendor SDK like Zipkin or Jaeger, the Collector can also receive those legacy protocols and re-export as OTLP, so you can move services over one at a time instead of doing a big-bang cutover.

Collector configuration

The otelcol-config.yaml below works as-is: three receivers (otlp for SDK data, hostmetrics for machine metrics, filelog for log files), all forwarded to JJHub through one OTLP HTTP exporter:

receivers:
  otlp:
    protocols: { grpc: { endpoint: 0.0.0.0:4317 }, http: { endpoint: 0.0.0.0:4318 } }
  hostmetrics:
    scrapers: { cpu: {}, memory: {}, disk: {}, network: {} }
    collection_interval: 30s
  filelog:
    include: [ /var/log/app/*.log ]
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<level>\S+) (?P<message>.*)$'

processors:
  batch: { timeout: 5s, send_batch_size: 512 }
  memory_limiter: { limit_mib: 512 }

exporters:
  otlphttp:
    endpoint: https://ob.jjhub.cn/otel
    headers: { "x-api-key": "${JJHUB_TOKEN}" }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [otlphttp] }
    metrics: { receivers: [otlp, hostmetrics], processors: [batch], exporters: [otlphttp] }
    logs:    { receivers: [otlp, filelog], processors: [batch], exporters: [otlphttp] }

The x-api-key header is the critical piece—generate it on the "Access Management" page of the console. Don't hardcode it into the repo; inject it via an environment variable so rotating the token only touches the environment. The batch and memory_limiter processors are the defaults you want: they smooth out bursts and protect the Collector from being overwhelmed.

Instrumenting the app

Java uses the javaagent for automatic instrumentation with zero code changes:

java -javaagent:opentelemetry-javaagent.jar   -Dotel.service.name=order-service   -Dotel.exporter.otlp.endpoint=http://localhost:4317   -Dotel.traces.exporter=otlp   -jar app.jar

HTTP, database, and message-queue calls automatically produce spans—no business code changes required. Set a service name per deployment, not per host, so a scaled service appears as one logical service in JJHub instead of N. In Go, manual instrumentation is more common; wrap your critical functions:

ctx, span := tracer.Start(ctx, "query_order")
defer span.End()

While you're at it, put business IDs (order number, user ID) into span attributes. That lets you look up an entire trace by order number during an investigation—business context that automatic instrumentation can't give you. Python and Node.js follow the same pattern: install the package, set the OTLP endpoint, and let the auto-instrumentation libraries handle the common frameworks.

Correlating logs with traces

For a log line to jump into its trace, it must carry a trace_id. In Java, inject it through Logback's MDC with one line:

<pattern>%d %-5level [%X{trace_id:-}] %msg%n</pattern>

In Go, write span.SpanContext().TraceID() into your log fields. Once both sides land in JJHub, any single log line can jump straight to its call chain—and the reverse holds too: from a span, you can open every log line that carries the same trace_id.

Common pitfalls

Three traps are worth knowing. Clock skew between containers and hosts breaks trace assembly, because span timestamps no longer line up—standardize on NTP. Over-sampling floods the backend, so keep trace sampling at 5%-10% in production while logs and metrics stay at full volume. And a missing headers config yields a 401, so verify the token with curl first, then check the app side, instead of guessing at both ends. If traces appear but logs don't, check that the filelog receiver's include path actually matches your log rotation pattern—rotated files often miss the glob.