Integrating OpenTelemetry with Ju Jing OBSERVE: From Instrumentation to Trace Search

Logs alone can't show you how a request flows through your services — for that you need traces. OpenTelemetry (OTel) is the de facto instrumentation standard, and wiring it into Ju Jing OBSERVE takes three steps: install the SDK, configure OTLP export, and confirm data lands in the console. Here's the full walkthrough with a Go service.

Step 1: Prerequisites

Before writing any code, confirm three things:

• Your Ju Jing OBSERVE is v2.0 or later, with the OpenTelemetry integration enabled in the console.
• The OTLP receiver (gRPC port 4317 or HTTP port 4318) is reachable from your network — open the port in the firewall.
• You've agreed on a service.name convention, like {domain}-{service}, so services are easy to find in trace search later.

Step 2: Install the SDK and initialize it

Start with the OTel SDK and the gRPC exporter in Go:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
)

func initTracer(ctx context.Context) (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("ob.jjhub.cn:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    res := resource.NewWithAttributes(semconv.SchemaURL,
        semconv.ServiceName("order-service"),
    )
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

Two things matter here. ServiceName is the dimension Ju Jing uses to separate services, so set it accurately per service. And WithBatcher batches spans before export, cutting network overhead roughly in half compared to sending each span individually.

Step 3: Automatic and manual instrumentation

For HTTP services, prefer automatic instrumentation — it's a one-line route wrapper:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux.Handle("/api/order", otelhttp.NewHandler(orderHandler, "place-order"))

Auto-instrumentation captures the "request received → response returned" span, but time spent inside the business logic — DB queries, downstream calls, computation — needs manual spans:

ctx, span := otel.Tracer("order-service").Start(ctx, "query-inventory")
defer span.End()
rows, err := db.QueryContext(ctx, "SELECT stock FROM inventory WHERE sku = ?", sku)
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

Always call span.RecordError and set codes.Error on the error branch. Otherwise error spans won't be marked red in the trace view, and you'll silently miss real failures.

Adding business attributes to spans is what makes traces searchable — don't just give a span a name. Put order IDs, SKUs, and regions in via span.SetAttributes so you can filter by business dimension later. But avoid high-cardinality attributes: they're persisted with the span and will slow queries if every span carries a unique value.

Step 4: Configure export and verify in the console

In the Ju Jing console, open "Integrations → OpenTelemetry" and generate an access token. Pass it as a header to distinguish tenants:

exporter, _ := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("ob.jjhub.cn:4317"),
    otlptracegrpc.WithHeaders(map[string]string{
        "Authorization": "Bearer " + token,
    }),
)

Once the service is up, send a few requests, then go to "Traces → Search" and query by service name or trace ID. You should see the full parent-child waterfall, the time share of each span, and the associated logs and alerts in the side panel. If nothing shows up after five minutes, check the exporter's error logs first — it's usually a refused connection or an expired token.

Step 5: Gotchas

• Clocks off. Make sure the host and the Ju Jing server are NTP-synced. Traces are sharded by time, and a few minutes of clock drift makes spans unqueryable.
• Sampling rate. Don't sample 100% on high-traffic services. Start with trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.1))) for 10%, then raise it once volume stabilizes.
• Span explosion. Don't create a span per loop iteration, especially in batch jobs — a single trace with tens of thousands of spans will freeze the search UI. Wrap the whole loop in one span instead.
• gRPC versions. The OTLP exporter and the server must agree on protobuf versions. When upgrading the SDK, check Ju Jing's compatibility notes to avoid the "connected but spans never land" situation.
• High-cardinality attributes. Don't stamp span.SetAttributes(attribute.String("user_id", uid)) onto every span indiscriminately. User IDs belong on key transaction spans only, or they'll bloat the index and slow trace search.

Step 6: Ship metrics through the same pipeline

Traces answer "where is this call slow"; metrics answer "is anything wrong overall." Use the same OTel SDK to export metrics too: initialize a MeterProvider in Go with go.opentelemetry.io/otel/sdk/metric, and add the runtime collector to report GC, goroutine count, and memory. Your service then has a performance baseline from day one, and when a trace shows a slow query you can check the metric dashboard to tell whether it's a one-off or a systemic degradation.

Finally, print the trace ID into your logs (log.Info("...", "trace_id", span.SpanContext().TraceID())) so logs and traces link together. From any log line you can then jump straight to the corresponding trace. Once logs, metrics, and traces are all connected, you're actually using observability.