Instrumenting Go Services with OpenTelemetry: A Manual Tracing Walkthrough

Why manual instrumentation matters

Auto-instrumentation covers the obvious calls — HTTP, databases, message queues — but it can't see your business logic: a tight loop, an external call, a block of encryption work. Manual instrumentation creates spans explicitly at those spots so the trace carries your business semantics. This guide builds a working instrumented Go service from scratch.

Initializing the SDK

Pull in the dependencies and initialize a TracerProvider that exports to a Collector:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) (*trace.TracerProvider, error) {
    exp, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exp),
        trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.1))),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

WithBatcher batches spans before export, and TraceIDRatioBased(0.1) samples 10%. In production, let auto-instrumentation own the framework entry points (HTTP, database) and use manual spans only to add business context.

Creating spans and attributes

A span is the basic unit of a trace. Thread the context through the call path and parent-child relationships are wired automatically:

func ProcessOrder(ctx context.Context, orderID string) error {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "process_order")
    defer span.End()

    span.SetAttributes(
        attribute.String("order.id", orderID),
        attribute.String("service.version", "1.4.2"),
    )
    // business logic...
    return nil
}

Be disciplined with attributes: only attach fields you'll actually search on — order id, user id, version. Don't stuff the whole request body in, or your storage and query costs will balloon. Events mark points in time, such as "payment callback received":

span.AddEvent("payment_callback_received",
    trace.WithAttributes(attribute.String("channel", "wechat")))

Marking errors

When a call fails, record it on the span as well as in the logs, so the trace graph flags it red at a glance:

if err := callInventory(ctx); err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

RecordError attaches the error as an event; SetStatus marks the span as ERROR. Upstream, filter on status = ERROR to pull every failed trace in one query.

Propagating context across services

A trace stays intact only if the context travels with the request. For HTTP, wrap handlers with middleware that injects the trace context into headers, and downstream services using the same SDK pick it up automatically:

otelhttp.NewHandler(handler, "order-api")

Async is where it breaks: in goroutines and message queues you must pass the context explicitly. Using a fresh context.Background() inside a goroutine severs the parent-child link. Sampling has the same trap — if a sampled parent spawns a child that makes its own sampling decision, you end up with half a trace.

Verifying the result

Fire a few requests and look up a trace by id in the console. Confirm each span's duration, attributes and error status are correct. Instrumentation isn't done when the code compiles; the completeness of the trace is only proven by a real request end to end.