Jujing OBSERVE v2.6.0: Prometheus Remote Write, Real-Time Log Pipelines, and Tail Sampling

New: Prometheus Remote Write support

Previously metrics came mainly through Jujing's own collector, so migrating an existing Prometheus stack meant touching the collection side. v2.6.0 natively supports Prometheus Remote Write — Prometheus, VictoriaMetrics, and Thanos can push metrics straight in with no changes to your exporters or scrape config. To onboard, add a remote_write block pointing at Jujing's receiver with your token:

remote_write:
  - url: http://metrics.jjhub.cn:8428/api/v1/write
    headers:
      Authorization: Bearer <your-token>
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'go_.*'
        action: drop

The write_relabel_configs is optional — use it to drop irrelevant metrics like go_* before they're pushed, saving bandwidth and storage. High-cardinality metrics are the usual culprit when Remote Write causes query timeouts downstream, so dropping them at the source is worth the ten minutes of config. Migrate gradually: let Prometheus dual-write (local storage plus Remote Write) for a week, verify the values match on both sides, then cut over cleanly. One note on the protocol: Remote Write is push-based, so metrics arrive faster than a scrape loop, but you give up Prometheus's local retention — keep that local storage as your source of truth until you're confident in the cutover. If you're already running Thanos or VictoriaMetrics, point their Remote Write at the same endpoint; the receiver treats them identically.

New: real-time log pipelines

Parsing used to rely on static rules in the collector, and changing a rule meant restarting it. v2.6.0 adds real-time pipelines that process the log stream before ingestion, with hot-reloaded rules and no restarts. A pipeline chains stages, each doing one thing:

pipeline:
  - stage: parse
    type: json
  - stage: mask
    fields: [card_no, id_number]
    method: regex
    pattern: '\b\d{16,19}\b'
  - stage: drop
    match: "level == 'debug' and env == 'prod'"

The first stage splits JSON into fields, the second masks card numbers and ID numbers by regex, and the third drops debug-level logs in production. Pipelines support parse, mask, filter, drop, add_label and more, so masking, filtering and labeling all happen before ingestion. For compliance-heavy teams, masking before ingestion means plaintext never hits disk — stricter than masking at query time, and the only option that satisfies some data-residency audits. Stages run in order, so put drop and mask early to save the downstream stages work. Two caveats: if a mask rule references a field that hasn't been parsed yet, that stage silently skips it, so order genuinely matters — test with a sample line before enabling a pipeline in production. And a dropped line is gone for good, so keep the drop conditions narrow until you've watched them in action.

New: tail-based trace sampling

Head sampling is random by ratio and discards many traces that only turn out to matter later — a request that starts normal but slows because a downstream hiccupped. v2.6.0 supports tail-based sampling: ingest all spans first, and once a trace completes, decide keep or drop. Configure keep-all for error traces, keep-all for traces over 2s, and 10% sampling for the rest. That caps storage cost without losing the slow requests and error chains that matter. Tail sampling runs as a tail_sampling processor on the collector and needs a small buffer — reserve 1–2GB of memory for it, and keep the decision window (default 30s) long enough that slow spans arrive before the trace is finalized. Compared with head sampling, tail sampling typically keeps 3–5x more useful traces per unit of storage, because it retains the rare, interesting ones deterministically rather than by luck.

Fixes and upgrade guide

Seventeen issues fixed: query timeouts from high-cardinality Remote Write metrics; pipeline parse failures on very large single log lines (over 1MB); boundary drift in the tail-sampling window across time zones. Upgrade: SaaS users are updated automatically; self-hosted users run upgrade.sh from v2.5.x, about 3 minutes with a brief alerting interruption. Back up alert rules and pipeline configs first, then verify the Remote Write endpoint and log pipelines after. Full changelog in the console's Release Notes.