v2.4.0 brings baseline-based anomaly detection, cross-rule alert grouping, log sampling, and ARM Xinchuang image optimizations. This post lists the key changes, config examples, upgrade steps, and compatibility notes.
OBSERVE v2.4.0 is out. This release focuses on three things: smarter alerting, cheaper storage, and a better Xinchuang experience. Here are the key changes and upgrade advice.
Until now alerts used fixed thresholds — "page when CPU exceeds 80%" — but load varies by time of day, so thresholds either false-alarm at night or miss peaks during the day. v2.4.0 adds baseline-based anomaly detection: the system learns the normal range from the same period over the past seven days and only alerts when the value deviates beyond three standard deviations. Example:
alert:
type: anomaly
metric: payment.latency.p95
window: 5m
sensitivity: 3 # 3 standard deviations
minDuration: 10mRun new metrics in parallel for two weeks first, comparing anomaly detection against the old fixed thresholds, and switch only after confirming no false alarms — don't replace production alerts on day one.
v2.4.0 introduces cross-rule alert grouping. Previously a single database blip triggered dozens of "dependency timeout" alerts; now the system merges alerts by root-cause dimensions (same service, same time window, same downstream dependency) into one, with a list of affected services and the common triggering path attached. The grouping window and dimensions are configurable:
alerting:
groupBy: [rootCause, downstream]
groupWindow: 5m
maxAlertsPerGroup: 100Teams with high log volume can now sample by rule: keep all ERROR logs, sample INFO at 10% and DEBUG at 1%, preserving the lines that matter for troubleshooting while cutting storage cost. Combined with the new hot/cold tiering, logs older than 30 days move to cold storage automatically; queries search across tiers in one click with no user-visible difference.
On the ARM side, official images now support Kunpeng 920 natively, the collector binary is built as AArch64, and eBPF collection is verified on Kylin V10. Three upgrade notes: first, the alert-engine config format changed, so migrate old rules to the new schema after upgrading; second, anomaly detection needs at least seven days of history before it activates, so there's no baseline immediately after upgrade; third, log sampling rules apply only to new data — existing logs are unaffected.
Plan the upgrade in two phases. Phase one: upgrade collectors everywhere and let them run for a few days against the old observer — the collector is backward compatible, so you can validate data flow before touching the backend. Phase two: upgrade the observer, starting with one node in the cluster and watching the alert engine under the new config schema.
Before you start, snapshot your alert rules. The v2.4.0 config format changes the alerting block; the migration tool converts most rules automatically, but rules using custom expressions or nested conditions may need manual review. Run the migration in a staging tenant first and diff the converted rules against the originals.
Anomaly detection needs seven days of history, so expect a quiet first week — that's by design, not a bug. To validate immediately, point a test metric with known spikes at the feature and confirm it fires. Rollback is straightforward: the observer supports downgrading one minor version, so keep the v2.3.x package around; collectors can stay on v2.4.0 without issue.
Beyond the headline features, this release also trims ingest CPU on ARM by roughly 12% through vectorized parsing, and fixes a leak that could stall the collector after days of eBPF probe churn. Teams already on v2.3 will feel the difference on Kunpeng clusters; everyone else gets the same alerting and sampling improvements with no schema migration beyond the alert engine.
Upgrade path: upgrade collectors first, then the observer, rolling — and don't jump major versions (anything below v2.2 must go through v2.2 first). See the Release Notes in the official docs for the full changelog.