OBSERVE v3.4.0: Visual Tail Sampling, Label-Based Alert Routing, and Slow-SQL Monitoring for Domestic Databases

OBSERVE v3.4.0 is released. This release makes concrete changes in trace sampling, alert routing, and domestic-database monitoring, driven largely by requests from teams running on domestic hardware. Full notes below.

This is a minor version with backward-compatible data formats, so existing dashboards, alert rules, and collector configs keep working unchanged. The two areas worth reviewing before you upgrade are the new alert routing and the index migration described below.

New features

• Tracing: visual tail-sampling configuration. Combine rules on status code, latency, and error rate, apply them from the UI without hand-writing YAML, and see the hit ratio of each policy reflected in a chart so you can verify the behavior against expectations. This removes the single most common source of misconfigured sampling: a policy ordering error that silently dropped the errors you meant to keep.
• Alerting: label-based grouping and routing. Alerts can be grouped by labels such as service or environment, and routed to different on-call teams, so no single team has to watch every service and each team's scope of duty becomes explicit. Routes are evaluated top to bottom, and the first match wins, which keeps the configuration predictable.
• Log search: regex and type-aware index pushdown. Filters on high-cardinality fields are pushed down to the storage layer; in testing, the P99 latency of these queries dropped by about 40%. This matters most for teams that search across many days of logs at once.

Together these three changes address the most frequent feature requests from the last quarter: sampling that engineers can verify at a glance, alert routing that matches how teams are actually organized, and search performance that holds up as log volumes grow. All three are enabled by default where they do not change existing behavior; the alert routing feature stays opt-in until you define your first route.

Improvements and fixes

• The collector uses less memory on Phytium and Loongson platforms, and CPU usage on ARM is lower still, which helps on small edge machines where every megabyte counts and where the collector runs alongside the workload it observes.
• Archival jobs for hot/cold tiered log storage now run concurrently per bucket, noticeably shortening startup latency for historical queries that touch old data, which previously had to wait on a single archive worker.
• Fixed tail sampling occasionally dropping an entire trace when spans arrived out of order. The buffer now reorders spans within the wait window before applying keep-or-drop rules.
• Fixed alert silence windows failing across day boundaries, such as 23:50 to 00:10 the next day. Silence windows now resolve correctly against a rolling time window rather than a naive date comparison.

Domestic-database enhancements

• Added slow-SQL collection and execution-plan drill-down for KingbaseES and openGauss, so you no longer have to rely on the database's own logs to find slow queries. Execution plans render inline in the trace and log views, which makes the jump from a slow request to a specific SQL statement far shorter.
• DM8 session metric collection now includes lock-wait counts and active transactions, giving you direct metrics for lock-contention investigation instead of inferring it from wait events in the database log.

Upgrade notes

Back up the metadata database before upgrading. This release includes index structure changes; during a rolling upgrade the collector protocol is backward compatible, so you can upgrade collectors before the server to avoid an ingestion gap. After upgrading, verify tail-sampling hit ratios and alert routing in a staging environment before rolling into production, and keep the previous version's config backed up so you can revert quickly if a sampling rule misbehaves.

If you run the platform on domestic hardware, note that the collector memory improvements apply automatically to new collector releases; upgrade the collector image to pick them up. For large installations, the index migration runs once on first start after the upgrade and can take several minutes per million documents, so plan a maintenance window rather than assuming a fast restart, and do not interrupt the migration once it begins. Rolling back within the same minor line is supported: if a sampling rule misbehaves after the upgrade, restoring the previous configuration and downgrading the server returns you to the prior behavior without re-ingesting any data.