OBSERVE v2.4: Parallel Log Aggregation, Alert Deduplication, and Metric Pre-aggregation

v2.4 focuses on the two things ops teams ask for most: making big queries fast and making alerts quiet. It also lands a batch of Xinchuang compatibility fixes and a new read-only query API. It ships in the standard rolling-upgrade flow with no data migration required. Here's what changed.

Log search: parallelized aggregation

Previously, cross-day GROUP BY aggregations scanned all shards serially, and windows over a day often took tens of seconds. v2.4 splits aggregation across data shards for parallel execution and merges results at the query node—measured 3-5x faster for 7-day aggregations, with common one-hour queries returning within about 2 seconds. It also adds the approx_percentile and approx_distinct functions, keeping error under 1% at 100k cardinality. No config changes needed; it takes effect automatically after upgrade.

Under the hood, the query planner now estimates per-shard cost and assigns work based on shard load, so one busy shard no longer holds up the whole query. For teams running multi-terabyte log stores, this is the release's biggest win: dashboards that used to time out now render in seconds. For teams that need exact percentiles, the legacy exact percentile() function remains available; the new approximate functions are simply the recommended defaults in the query builder.

Alerting: root-cause grouping and escalation

Alert noise is the most painful part of on-call, and a single database hiccup can fire dozens of alerts. v2.4 adds two layers of deduplication. First, root-cause grouping: sustained jitter on the same service and metric merges into one alert carrying the list of affected hosts and instances. Second, escalation: alerts that stay unresolved can be auto-promoted (e.g., P3 to P2 after 30 minutes), with silence windows supported.

alert:
  group_by: [service, metric]
  escalate:
    - from: P3
      to: P2
      after: 30m

Grouping is configurable, and the order of the group_by keys matters: [service, metric] merges all hosts of a service into one alert, while [service, host, metric] keeps per-host alerts separate. Escalation actions reuse the same notifiers as alerts—SMS, phone, webhook, and the in-network notifiers adapted for Xinchuang environments. After rollout, observe for a week, then tune group_by against your real jitter patterns so alerts with different root causes don't get merged incorrectly.

Metrics: pre-aggregation and long-term storage

Previously, querying metrics older than 30 days required reading raw data—slow and resource-hungry. v2.4 adds a pre-aggregation engine that rolls raw metrics into 1-minute, 1-hour, and 1-day granularities at write time, and queries auto-select the best granularity by time span. Combined with new downsampling policies, data older than 90 days can retain only daily granularity, cutting storage costs by roughly 60% without losing the ability to see annual trends.

Existing metric data is backfilled into the new granularities automatically during the rolling upgrade, so long-range charts have no "blank period." If you want stricter retention, you can set per-metric overrides for high-cardinality series that are expensive to keep. The pre-aggregation engine runs as a separate service (jj-metric-rollup) that you can scale independently when backfilling a large historical dataset. A typical downsampling policy:

downsampling:
  - after: 90d
    keep: 1d
    except: ["business_error_rate"]

Raw 1-minute data is kept for 30 days by default; set keep to a coarser granularity for anything older, and use except to pin specific metrics to full resolution when regulators or SLAs require it.

Other changes and upgrade notes

Xinchuang: official support for UOS 20 SP3 and DM8, and a fix for off-heap allocation errors in the Java collector on ARM64. Security: query audit logs now record field-level detail, and all alert-rule and silence-window changes are versioned. API: a new /api/v1/query endpoint exposes the same SQL engine used by the console, so CI pipelines and scripts can run aggregations programmatically. Backward compatibility: v2.4 is fully compatible with v2.3 config files, and the query language is a strict superset—nothing written for v2.3 breaks. One behavior change to note: alert grouping now defaults to [service, metric] instead of no grouping, which may reduce alert volume for existing rules; review your top-volume rules after upgrading. Upgrades: in-place upgrade.sh works from v2.3 and later—back up the metadata database first; data shards upgrade in rolling fashion without interrupting collection. See the release notes for the full list, and file issues via Console → Help → Submit Ticket.