Wiring OpenTelemetry into JJHub OBSERVE: From Instrumentation to Full Tracing

Get the smallest loop working first

The most common OpenTelemetry mistake is rewriting a lot of business code up front and then getting stuck on exporter configuration. Start with a minimal loop - instrument, export, query - and expand from there. The loop has three stages: your app emits spans, the exporter ships them to OBSERVE, and you see a trace in the UI. Once that works end to end, scaling up is mostly a matter of enabling the same thing in more services.

For a Java service, the javaagent gives you auto-instrumentation with no code changes:

java -javaagent:opentelemetry-javaagent.jar   -Dotel.service.name=order-service   -Dotel.exporter.otlp.endpoint=http://ob.jjhub.cn:4318   -Dotel.traces.exporter=otlp   -Dotel.metrics.exporter=otlp   -Dotel.logs.exporter=otlp   -jar order-service.jar

The agent automatically creates spans for common components like Spring MVC, MySQL clients, and HTTP clients, so you don't hand-write instrumentation for framework boundaries. For Go, the same applies - pull in the otel instrumentation packages for gin, gorm, and so on. The service.name you set is the identity that appears across traces, metrics, and the services list, so pick a stable name and keep it consistent across environments. If you prefer a config file over environment variables, the same settings go into otel-config.yaml under the OTLP exporter block - handy when managing many services from one version-controlled config.

A word on context propagation: the traceId travels in HTTP headers and message metadata, so it flows across service boundaries and even async queues automatically once each hop is instrumented. If a trace breaks at a particular boundary, the first thing to check is whether that hop propagates context - a manual HTTP client or a queue consumer that drops headers is the usual culprit.

Add manual spans on the critical path

Auto-instrumentation covers framework boundaries, but the business steps inside - placing an order, deducting inventory, calling a third-party payment gateway - need manual spans to become visible. Without them, a slow order looks like "the HTTP handler was slow" instead of "the payment call inside it took 900ms".

Span span = tracer.spanBuilder("place_order")
    .setAttribute("order.id", orderId)
    .startSpan();
try (Scope scope = span.makeCurrent()) {
    inventoryService.deduct(orderId);
    paymentService.pay(orderId);
} finally {
    span.end();
}

Attach meaningful attributes - order id, user id, channel - so you can filter traces by business object later, not just know which service was slow. A span with an order_id is what lets support staff paste a customer's order number into the trace view and get the full path back. The same pattern applies in Go with tracer.Start(ctx, ...): the API differs slightly, but the idea of wrapping a business step in a named span with attributes is identical.

Configure OTLP export and sampling

JJHub OBSERVE accepts both OTLP/HTTP (port 4318) and OTLP/gRPC (port 4317). In production, prefer gRPC with gzip compression to cut network overhead; HTTP is convenient for quick local tests because you can hit it with curl. Always include the protocol prefix in the endpoint - a bare host:port is a frequent source of "export timed out".

Sampling controls both cost and diagnosability, so pick per scenario:

• Core transaction paths: 100% sampling, keep everything for post-hoc review
• Ordinary endpoints: tail-based sampling - low rate normally, capture everything on errors
• High-volume logs: export error level and above only

Sampling is a trade-off, not a set-once setting. Tune it against your retention budget and revisit when traffic patterns change - a sampling rate that was fine at 10 rps may flood your storage at 1,000 rps.

Verify and troubleshoot

After wiring it up, verify in this order:

1. Trigger a request and filter the trace view by service.name - a trace should appear
2. Open the trace and confirm span timing and attributes are intact
3. Check the Services page to see the service auto-discovered with QPS and latency metrics

Common issues and their likely causes: export times out because the endpoint lacks the protocol prefix; traces are missing because the sampling rate is zero; spans lack attributes because startSpan was never called; timestamps look shifted because the app and OBSERVE aren't both on NTP. Without a local OBSERVE instance, route through an otel-collector by pointing the endpoint at it and forwarding from there - the collector also lets you batch, retry, and redact data before it leaves your network.