Onboarding the OpenTelemetry Collector: One Pipeline for Logs, Metrics and Traces

Why use a Collector instead of direct app connections

Pointing applications straight at the observability platform is the simplest setup — until you have scale. Dozens of services each pushing data to the platform means configuring a token, an endpoint, retries and backpressure for every single one; change the ingestion URL and every service needs a config update. Worse, logs, metrics and traces often come from different libraries with inconsistent fields, formats and sampling policies, so the platform receives a mess it has to normalize downstream.

The OpenTelemetry Collector is a middle layer: applications send data only to a local Collector, which handles processing, filtering, batching and exporting in one place. The application side only worries about one thing — send to the Collector — and everything else is the Collector's job. This also means you can swap the backend (change vendors, or add a second one) by editing a single exporter, without touching any application. In short, the Collector turns a fan-out of fragile app-to-platform connections into one managed pipeline.

Deploying a Collector

Start with a single-node Collector running in Docker. Two receivers are defined here: otlp for SDK-instrumented applications and filelog for tailing raw log files:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  filelog:
    include: [ /var/log/app/*.log ]
    start_at: end

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlphttp:
    endpoint: https://ob.example.com/v1/traces
    headers:
      Authorization: "Bearer <token>"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

A quick note on the pieces: receivers define where data comes in, processors transform it, exporters define where it goes out, and service.pipelines wires them together. The batch processor groups small packets into larger ones to cut the number of network round-trips, and memory_limiter keeps the Collector from exhausting its own memory under load. You can add more processors — for example a filter to drop debug logs before they leave the host, which saves ingest volume and money.

How to route the three signal types

Observability data comes in three signals, with separate export endpoints:

• traces: /v1/traces, distributed tracing
• metrics: /v1/metrics, metrics
• logs: /v1/logs, logs

In service.pipelines, create one pipeline per signal. Receivers can share the same OTLP port, or you can dedicate different ones per signal if you want independent buffering and retry behavior. A common pattern is one OTLP receiver feeding three pipelines, each with its own exporter endpoint. Splitting the signals is worthwhile because each has different cardinality and volume, and you'll often want different batch sizes for logs (high volume) versus traces (low volume but latency-sensitive).

What changes on the application side

After the application adds the OpenTelemetry SDK, point its OTLP endpoint at the local Collector (localhost:4317). You no longer care about the platform's URL and token — those are configured once, in the Collector. This is the real payoff: onboarding a new service becomes "add the SDK and set one environment variable," and the platform credentials never leave the Collector host.

For Java, inject them via environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=order-service

Pitfalls you will hit

• Always set memory_limiter: under high concurrency the Collector's memory grows fast; without a limit it OOMs and drops everything in flight. Set limit_mib to a fraction of the host's memory and spike_limit_mib for bursts.
• Don't set send_batch_size too small: tiny batches mean frequent requests that slow things down; around 8192 is stable for most workloads. Too large, and you add latency waiting for the batch to fill.
• Put the token in headers: the OTLP HTTP exporter uses Authorization: Bearer <token>, not a URL parameter. Getting this wrong is the most common cause of "data isn't showing up."
• Unify log timestamps: when filelog tails files, confirm the timezone, or every log timestamp shifts together and your time-based queries silently miss data.

Once connected, verify all three signals landed with a single query in the Observe console:

SELECT source, count(*) FROM observability
WHERE ts > now() - 15m GROUP BY source

All three present means onboarding is complete. If a signal is missing, check its pipeline in the Collector config first — the receiver, processor and exporter must all be wired in service.pipelines, which is easy to forget.