Shipping Prometheus Metrics to OBSERVE: remote_write Setup and Three Gotchas

Why push metrics to OBSERVE

Plenty of teams already run Prometheus for metrics, so standing up a second set of OBSERVE agents is duplicated work. remote_write is Prometheus’s native export path — a few lines of config streams metrics into OBSERVE while the old monitoring keeps running. Lowest risk, and you keep both stacks side by side during the transition.

OBSERVE’s metrics endpoint speaks the Prometheus remote_write protocol and understands PromQL, so teams that migrate barely have to touch their dashboards or alert rules. That’s the point: you shouldn’t have to relearn query syntax just to move data.
Under the hood, remote_write batches samples into the WAL, fans them out across shards, and pushes them over HTTP with protobuf. That's why it's the lowest-risk migration path: Prometheus keeps scraping and alerting locally, and OBSERVE simply becomes a second, always-in-sync copy of the same data.

Basic setup

Add a remote_write block to prometheus.yml:

remote_write:
  - url: https://ob.jjhub.cn/api/prom/remote_write
    headers:
      Authorization: Bearer <token>
    remote_timeout: 30s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*|process_.*'
        action: drop

Generate a dedicated write token per project in the OBSERVE console — don’t reuse the admin token — so you can revoke access per project later without breaking everything else. The relabel rule above also drops Go runtime and process metrics you probably don’t need at the remote end, which saves bandwidth from day one. Restart Prometheus and the new data source shows up in the OBSERVE metrics page.

Tuning the queue

Internally remote_write runs a WAL plus a send queue. Misconfigure it and you get one of two classic failures: dropped samples or ballooning memory. Three parameters matter most:

• capacity — the in-memory queue length per shard, default 2500. If Prometheus logs queue full under load, raise it, say to 10000.
• max_shards — the concurrency ceiling for sending, default 200. Raise it when the remote network is slow, and watch CPU while you do.
• max_samples_per_send — samples per batch, default 500. On high-latency links, raising it to 2000 improves throughput noticeably.

A conservative setup for a mid-size workload (a few million samples per minute):

queue_config:
  capacity: 10000
  max_shards: 50
  min_shards: 2
  max_samples_per_send: 2000
  batch_send_deadline: 10s
  min_backoff: 100ms
  max_backoff: 10s

Don’t validate tuning by reading docs — watch two metrics instead: prometheus_remote_storage_samples_failed_total (failed samples) and prometheus_remote_storage_highest_timestamp_in_seconds (latest pushed timestamp). A steadily climbing failure count means the queue is misconfigured; a stuck timestamp means the channel is down. These two numbers tell you more than any guide.

Three gotchas

Gotcha 1: high-cardinality write amplification. A metric carrying user IDs or request IDs as labels explodes the series count, and a single mislabeled metric can drag down the whole channel. Drop high-cardinality labels at the source:

write_relabel_configs:
  - source_labels: [user_id, request_id]
    regex: '.+'
    action: labeldrop

Gotcha 2: silent loss when the remote is down. Prometheus queues failed samples for retry, but the WAL fills up if OBSERVE is unreachable for a long time. Watch prometheus_remote_storage_shards_desired; if it keeps hugging max_shards, the remote can’t keep up and you need to scale it or downsample.

Gotcha 3: leaked tokens. remote_write sends the token in a plain HTTP header, so never commit prometheus.yml to a public repo. Keep the token in an environment variable or a Secret, and rotate tokens on the OBSERVE side periodically. A leaked write token is a backdoor into your monitoring data.

Label strategy: what to keep and what to drop

Remote write copies every label you send, and the labels you keep become your query dimensions forever, so think before you send. A good rule of thumb: keep labels that describe what you'll group by — service, instance, region, endpoint — and drop anything that identifies a single request or user. Those belong in traces, not in metrics.

One more practical tip: if you run Prometheus in an HA pair, stamp each replica with a unique label before it leaves, so you can tell the two apart at the remote end instead of seeing double-counted series.

Verification checklist

After wiring it up, check in order: the data source appears on the metrics page → a sample PromQL query returns data → the sampling rate matches the source → the failed-sample count is zero → no high-cardinality alerts fire. Pass all five and you can start migrating alert rules off the old stack without losing a beat.