Observability for IoT Edge Gateways: Weak Networks, Offline Buffering, and Batch Reporting

Edge gateways sit in factory floors, server rooms, and job sites with unreliable networks and limited disk and memory, so logs and metrics cannot be pushed in real time the way cloud services can. The core idea is simple: buffer locally on the edge, then report upstream in batches with compression and resume support. Done right, this keeps data loss near zero even through multi-day outages without overloading the gateway.

Edge side: local buffering plus batch reporting

On the gateway, run an OpenTelemetry Collector and use the file_storage extension to write data to local disk first, then forward to the center asynchronously:

extensions:
  file_storage:
    directory: /var/lib/otelcol
    timeout: 1s

exporters:
  otlphttp:
    endpoint: https://ob.jjhub.cn/otel
    sending_queue:
      storage: file_storage
      queue_size: 10000

During an outage data lands on disk first, then the queue replays once connectivity returns, so nothing is lost. Size queue_size against the gateway's disk: multiply one day of data by the longest expected outage and make sure the disk never fills. A common rule of thumb is to reserve enough capacity for three days of buffering so a long weekend outage does not silently drop data.

Set a retention watermark so the queue never fills the partition: when buffered data exceeds the watermark, the collector drops the oldest data rather than stalling the pipeline. For gateways that cannot afford any loss, raise the watermark and add a larger disk; for telemetry that is only sampled aggregates, a lower watermark is perfectly acceptable.

Compression and sampling on weak networks

Edge-to-center bandwidth is often only a few Mbps or less. Enabling gzip on OTLP HTTP saves most of the traffic; aggregate metrics on the edge every 30 to 60 seconds before reporting; keep only error-level logs and key business events while rolling away ordinary info logs locally. That controls both bandwidth and the storage cost on the center side. Also set a conservative retry backoff on the exporter so that a flapping link does not hammer the center with reconnect storms, which on a shared low-bandwidth link can starve every other device on the site.

A second bandwidth lever is to send metrics and logs on different cadences. Metrics are small and time-sensitive, so push them every 30 seconds; logs are bulky and only interesting during an incident, so buffer them for minutes and flush in larger batches. Splitting the two avoids a burst of error logs blocking metric updates just when you need the metrics most.

Device identity and multi-tenant isolation

Give every gateway a unique service.name and resource attributes such as site_id and gateway_id, and build dashboards and alerts per site on the center side. Use site_id as the grouping key in alert rules so one gateway going offline only affects itself instead of spamming the on-call channel.

receivers:
  otlp:
    protocols: { http: {} }
processors:
  resource:
    attributes:
      - key: site_id
        value: "factory-3"
        action: upsert

Keep the attribute set small and consistent across the fleet. When every gateway reports the same key names, a single center-side dashboard template can be reused for all sites instead of being rebuilt per site. For security, put a TLS certificate on each gateway and rotate it as part of the normal fleet lifecycle, so telemetry from a compromised device cannot be silently spoofed.

The debugging path

When a gateway alert reports interrupted upload, first check the last-report timestamp on the center dashboard to tell a network problem from a device problem, then log into the gateway and watch the file_storage directory growth and the collector's queue metrics. The most common root cause is a full disk deadlocking the queue, so give /var/lib/otelcol its own partition and a retention watermark; that saves far more pain than firefighting afterwards. Monitoring the collector itself, including its queue depth and disk usage, turns a vague offline alert into an actionable diagnosis, and it means the first responder can tell within minutes whether the gateway is healthy and the link is the culprit.

Finally, treat the gateways as cattle, not pets: bake the collector config, certificates, and site_id into a signed image so a failed gateway can be swapped in minutes and automatically reports under the same identity. The observability story falls apart if each replacement gateway arrives with a fresh, unknown identity and a blank dashboard.