OBSERVE v2.5: Alert Aggregation, Trace Sampling, and Domestic-Stack Support

Overview

v2.5 focuses on three areas: alerting experience, trace storage cost, and domestic-stack support. It adds alert aggregation and silence windows, multiple trace sampling strategies, and completes adaptation and verification for Loongson LoongArch and Dameng DM8. The release is backward-compatible with v2.4 data formats, so upgrades require no downtime. Every feature below is opt-in except the alert-config migration, which happens automatically. As with every minor release, we recommend upgrading staging first and letting it run a full business cycle before promoting to production. The 2.4.x branch will continue to receive critical bug fixes through the end of this year, so there's no rush to move if you're mid-release-cycle.

Alert Aggregation and Silence Windows

Previously, a rule firing 20 times sent 20 notifications, flooding the on-call phone and burying the one message that mattered. v2.5 adds alert aggregation: triggers from the same rule and service within a window merge into one notification, with the trigger count and first/last trigger times. The notification also links to the full trigger list, so nothing is lost—just deduplicated at the delivery layer.

It also supports time-based silence: alerts are automatically suppressed during release windows or planned maintenance, then restored automatically, so a routine deployment no longer wakes anyone at 3 a.m. Configuration example:

alert:
  aggregate:
    window: 5m
    by: [rule_id, service]
  silence:
    - start: "02:00"
      end: "04:00"
      match:
        service: "batch-job"

Trace Sampling Strategies

Trace storage cost has long been a pain point in high-traffic scenarios. v2.5 supports three sampling modes: fixed-ratio sampling, error-first sampling (100% of failed requests retained), and span-duration sampling (all slow requests retained). They can be combined—for example, 10% sampling for normal traffic plus full retention of errors and slow requests—cutting storage cost by over 60% while keeping problem traces fully inspectable.

tracing:
  sampling:
    ratio: 0.1
    keep_errors: true
    keep_slow: 2000ms

Sampling is evaluated at the collector, so downstream dashboards and alerts automatically reflect the reduced volume. The sampled-out ratio is reported as a metric, which makes it easy to confirm the policy is actually in effect rather than assuming it is.

Performance Improvements

Alongside the feature work, v2.5 ships a batch of performance improvements. Log search over large result sets now streams pages incrementally, so the first page renders in under a second even when the full result spans millions of rows. Dashboard queries that cross midnight boundaries are bucketed correctly, fixing a long-standing off-by-one that skewed overnight charts. And on ARM nodes, several monitoring collectors had their buffer handling reworked, cutting steady-state memory by roughly a third.

Domestic-Stack Support

v2.5 completes compilation and full stability testing for the Loongson LoongArch architecture, and verifies Dameng DM8 as the metadata database for backup/restore and compatibility with Kylin V10 security patch upgrades. The LoongArch build passes the same end-to-end test suite as the x86 release—ingest, query, alerting, and failover—so teams on Loongson hardware can treat it as a first-class platform rather than a compatibility afterthought. Images for Hygon, Kunpeng, and Phytium architectures are updated to 2.5 as well.

Deprecations and Upgrade Notes

Two things are deprecated in v2.5. The legacy webhook notification format (/v1/notify) is superseded by the v2 payload that carries aggregation metadata; it still works but will be removed in v2.7, so migrate your webhook handlers now. The old --single-node deployment mode is also deprecated in favor of the standard three-node topology, which is now supported on all domestic architectures with no performance penalty.

Also fixed in this release: pagination performance in log search over large result sets, a boundary bug in dashboard time ranges spanning midnight, and higher-than-expected memory usage in some monitoring collectors on ARM. Upgrades are script-driven and support rolling updates with no perceived downtime:

./deploy.sh upgrade --version 2.5.0

Note that v2.5 adjusts the alert configuration format; old rules are migrated automatically, but export a backup before upgrading. Release artifacts are available as architecture-specific images and offline bundles for air-gapped environments—the offline bundle includes all dependent images and the metadata-database schema scripts, so a fully offline upgrade is a two-command operation. See the "Version History" page in the docs for the full changelog.