Auto-instrumentation misses business logic, so manual spans are the only way to see your own code paths. Using a Go service as an example, this guide walks through creating spans, propagating trace context, recording errors and retry events, adding metrics, and verifying the result.
Auto-instrumentation covers the HTTP, gRPC, and database-driver calls the framework makes, but it can't see your own business logic: which steps a single order goes through between creation and stock deduction, how long each step takes, which one retried three times. Only the person writing the code knows that. If you want to answer "why is this order slow," you have to instrument the critical path yourself. And don't instrument everything—start with the chains people complain about most or that break most often; that's where the payoff is biggest. And instrument early: retrofitting spans after a few incidents and several rounds of complaints costs far more, while adding the key spans as a feature ships is the cheapest path.
For a Go project, pull in the dependencies first:
go get go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttpThen initialize a TracerProvider that reports to Observe over OTLP/HTTP:
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func initTracer(ctx context.Context) (*sdktrace.TracerProvider, error) {
exp, err := otlptracehttp.New(ctx,
otlptracehttp.WithEndpoint("ob.example.com:4318"),
otlptracehttp.WithInsecure(),
)
if err != nil { return nil, err }
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))
otel.SetTracerProvider(tp)
return tp, nil
}Two things to fix for production: switch to HTTPS, and set the service.name and deployment.environment resource attributes per instance—otherwise you can't tell machines or environments apart once the data lands in the platform. Inject resource attributes with resource.NewWithAttributes when you build the provider, and they apply globally. Heavy-traffic services should also configure sampling: add sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))) on the provider to keep 10% of normal requests but the full trace whenever an error occurs, balancing cost against debuggability. Changing the ratio needs no code change—just a restart.
The one rule that matters: pass context down. Take order creation as an example, wrapping stock deduction and order persistence each in their own span, with the attributes you'll actually query on:
func CreateOrder(ctx context.Context, o *Order) error {
tracer := otel.Tracer("order-service")
ctx, span := tracer.Start(ctx, "create_order")
defer span.End()
span.SetAttributes(
attribute.String("order.id", o.ID),
attribute.Int("order.items", len(o.Items)),
)
ctx, stockSpan := tracer.Start(ctx, "deduct_stock")
err := deductStock(ctx, o)
stockSpan.SetAttributes(attribute.Int("stock.deducted", o.TotalQty))
stockSpan.End()
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "deduct stock failed")
return err
}
return nil
}A few conventions worth following: name spans with verb phrases (deduct_stock, not span1); mark failures with RecordError plus SetStatus so you can filter straight to failed traces later; and only add attributes you'll actually search on—don't dump the whole request body, it wastes storage and makes traces unreadable.
Beyond latency, spans can record "what happened." If stock deduction retries internally, capture it with AddEvent so you can see the struggle without digging through logs:
for i := 0; i < 3; i++ {
if err := deductStock(ctx, o); err == nil { break }
stockSpan.AddEvent("retry", attribute.Int("attempt", i+1))
}Events keep only a few attributes by design—perfect for discrete facts like "retry attempt 2" or "degraded to replica B," and the wrong place to dump large log chunks.
The mistake people make most often with manual instrumentation is dropping the context. A trace becomes a single chain only because the same trace context rides along through ctx: database calls and downstream HTTP requests must use the ctx you wrapped, not a fresh context.Background(). Otherwise every span sits in isolation and you can never reconstruct the full path. If an internal function doesn't take a context today, add one—it's a two-line change that keeps the trace from breaking there. The same goes across processes: when you publish to a message queue or call a downstream HTTP service, put the trace context into the message or request headers—otherwise the consumer starts a brand-new trace and the two halves never line up.
The same SDK can emit metrics to power alerts. Attach a histogram to order latency:
import "go.opentelemetry.io/otel/metric"
orderLatency, _ := meter.Float64Histogram("order.latency",
metric.WithUnit("ms"))
func CreateOrder(ctx context.Context, o *Order) error {
start := time.Now()
defer func() {
orderLatency.Record(ctx, float64(time.Since(start).Milliseconds()))
}()
// ...
}Traces answer "why was this one slow"; metrics answer "is it slow overall." Give both the same attributes (service, order.id, and so on), and you can drill from an aggregated alert down to a single trace—that's what closes the troubleshooting loop.
Start the service, place a test order, then search the trace view in Observe by order.id. You should see the create_order → deduct_stock parent-child structure, per-span latency, and the error status and retry events on the failed attempt. Cross-reference the trace_id from your logs and you can pin "why is order 12345 slow" to a single step. A quick checklist: do all spans sit under the same trace? Any flat spans with no parent? Can you search attributes in the query panel? Do failed traces filter by status=error? If spans show up flat, context propagation is the first thing to check. And wire all of this up in staging, not production—when instrumentation problems mix with real business bugs, it's hard to tell whether the code is wrong or the observation is wrong.