OBSERVE v2.8.0: Alert Root-Cause Analysis and Natural-Language Log Search

Overview

v2.8.0 delivers the two things promised in the v2.7.0 roadmap: alert root-cause analysis and natural-language log search. It also strengthens distributed tracing and ships a batch of Xinchuang and stability fixes. This release focuses on the two areas that eat the most time during incidents — figuring out what broke and finding the relevant logs — so the headline items are both about faster investigation rather than new data sources. Read "Upgrade notes" before upgrading.

New features

• Alert root-cause analysis: when an alert fires, the platform automatically correlates it with the nearest anomalous metrics, error logs, and slow traces, ranking them by "suspicion score." Mechanically: within the alert's time window, it aggregates error-rate spikes, error-log signatures, and P99-latency traces for the alerted service, then computes each candidate's contribution. It doesn't make the decision for you, but it shrinks "where do I start looking" from ten-plus minutes to seconds. Each candidate carries an evidence trail — which error logs and slow traces it links to — and every item is clickable through to the raw data, so you can verify the ranking yourself. It also learns from which candidates you click, so the ordering improves as your team uses it;
• Natural-language log search: the search box accepts a plain-language request and translates it to SQL. Typing "5xx errors on the payment API in the last hour, grouped by user" produces the matching query, and the translation is shown for confirmation before running — no black-box query surprises. The translation runs against your actual schema and field names, so it resolves to queries that work on your data;
• Tracing improvements: new span-level custom sampling (keep high-value traffic by service.name and attribute), and service topology now toggles coloring across QPS, error rate, and latency;
• One-click alert-rule cloning: duplicate an existing rule and tweak the threshold instead of re-entering everything. Cloning keeps the full rule definition intact and lets you edit the threshold in place, which is handy when you need "same rule, different service."

Performance improvements

• The semantic parser for natural-language-to-SQL runs on a local small model, averaging about 40ms per translation, with no external network dependency; the model is quantized and runs CPU-only, so no GPU is required on the platform host;
• Root-cause analysis runs asynchronously after an alert fires, so it never blocks notification; 100 concurrent alerts return results in about 3 seconds on average;
• The high-cardinality GROUP BY path in log search was optimized, cutting query latency for thousand-group aggregations by roughly 35%.

Bug fixes

• Fixed DM database intermittently reporting "column does not exist" when reading alert rules under case-sensitive mode;
• Fixed eBPF flame graphs slowly leaking memory on Kylin V10 during long runs;
• Fixed cross-tenant alerts notifying the wrong tenant under specific label combinations (security fix — upgrade promptly);
• Fixed OTLP log trace correlation breaking when sampling dropped some spans;
• Fixed multiline log parsing occasionally dropping the first line at file-rotation boundaries.

Upgrade notes

1. Back up config and database before upgrading; no index format changes, so no reindexing is required;
2. Root-cause analysis depends on trace_id correlation — confirm both logs and spans carry trace_id before enabling, or ranking degrades to "metrics only";
3. Natural-language search is off by default; enable it in system settings. The local small model needs about 200MB on first load — prepare an offline package for air-gapped environments;
4. New config keys alert.root_cause.enabled and nl2sql.model.path control root-cause analysis and the semantic parser respectively;
5. After upgrading, run a smoke pass: fire an alert and check root-cause analysis, run a natural-language query, and confirm the trace sampling policy takes effect.

Other changes

• Removed compatibility with the long-discontinued Filebeat 5.x ingest protocol; environments on old protocols should upgrade their Agent first;
• Default log retention moved from 90 to 180 days to align with graded-protection compliance; change it back in config if you need;
• The docs site adds usage guides for root-cause analysis and natural-language search.

Known issues

• Natural-language search has limited support for colloquial Chinese (like "帮我看下" / "help me check"); describe queries as "metric + filter + aggregation" for the most reliable results;
• Root-cause analysis is weaker at "external dependency timeout" root causes (e.g. a third-party API outage); external-dependency profiling lands in a later release;
• Span custom-sampling rule changes only affect new data, not already-stored history.

Roadmap

v2.9.0 planning starts now, prioritizing two items: external-dependency profiling for root-cause analysis, and search-to-alert (save frequent searches as rules that page on anomalies). Vote in the release notes' issue tracker.