Unified Ingestion with the OpenTelemetry Collector

Why you want a Collector

Many teams integrate with an observability platform the hard way: Logback appenders for logs, Micrometer for metrics, the OTel SDK for traces — each wired separately. Once the service count grows, these three configurations become a maintenance burden, and changing one ingestion endpoint means touching dozens of services. Worse, each path applies its own sampling, buffering and retry logic, so the three signals drift out of sync and become hard to correlate during an incident.

The OpenTelemetry Collector collapses all of that into a single point. Applications only send telemetry to the Collector, which handles filtering, redaction, routing and export uniformly. To switch backends, you change the Collector once. Beyond consolidation, the Collector gives you one place to enforce sampling, redaction and rate limiting — policies that would otherwise be re-implemented, slightly differently, in every single service.

Four concepts that tie the config together

The Collector is organized around four concepts, and understanding them makes the config file far less intimidating:

• Receivers pull or accept telemetry — OTLP over gRPC/HTTP for traces and metrics, a filelog receiver for log files.
• Processors transform data in-flight — batching, memory limits, redaction, filtering.
• Exporters send data to a backend — here, the OTLP HTTP exporter pointed at Observe.
• Pipelines connect them: each signal type gets its own receivers → processors → exporters chain.

That layering is why one Collector can serve logs, metrics and traces at once, each with different processors.

Deploying the Collector

Run a single instance in agent mode — a binary or container on the same host (or Pod) as your services — so telemetry is gathered locally before export:

docker run -d --name otelcol \
  -v ./otelcol-config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector-contrib:0.95.0

Here is a skeleton that ships all three signal types to Observe:

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  filelog:
    include: [/var/log/app/*.log]
    operators:
      - type: regex_parser
        regex: '^(?P<ts>[^ ]+) (?P<level>\w+) (?P<msg>.*)$'

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlphttp:
    endpoint: https://ob.example.com/otlp
    headers:
      Authorization: "Bearer ${OBSERVE_TOKEN}"

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp] }
    logs:    { receivers: [otlp, filelog], processors: [memory_limiter, batch], exporters: [otlphttp] }

Three pipelines share one otlphttp exporter, so logs, metrics and traces are configured in a single pass. For larger fleets, add a second Collector in gateway mode to aggregate several agents before forwarding to Observe — but start with agent mode.

What changes in your application

Applications no longer talk to the platform directly; they point at the local Collector on port 4317:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=order-service
java -javaagent:opentelemetry-javaagent.jar -jar app.jar

Java, Go and Python all ship agents or SDKs; once attached, logs, metrics and traces flow over OTLP automatically. If an app cannot add a dependency, the filelog receiver above can still tail log files and forward them as structured records.

Do redaction and filtering here

The Collector is the best place for unified data governance. Production logs are full of phone numbers and ID card numbers, so add a redaction or transform processor to strip sensitive fields before they leave the machine — not after the data is already in the platform:

processors:
  redaction:
    allow_all_keys: false
    blocked_values: ["\d{11}", "\d{18}"]

Doing this at the edge means a misconfiguration never leaks raw sensitive values into the platform or its backups.

Verify the data landed

After the app starts, confirm each signal shows up in Observe:

SELECT * FROM logs WHERE service = 'order-service' ORDER BY ts DESC LIMIT 10

Then open a recent request in the trace list and confirm the spans nest correctly. If logs arrive but traces do not, the trace pipeline is usually missing its receiver, or the exporter is misconfigured.

Troubleshooting checklist

• No data arriving: curl http://localhost:4318/v1/traces to confirm the Collector is up.
• Only one signal type: check whether the matching pipeline is missing its receiver.
• Getting 401s: verify the Authorization header token matches the one in the platform's Service Management.
• Timestamps off: standardize on UTC or set the timezone explicitly so trace and log timestamps don't drift apart.