OBSERVE v2.6.0: Faster Log Search and a New Trace Flame Graph

Overview

v2.6.0 focuses on three things: faster log search, stronger tracing, and a more stable collection Agent. Read the "Upgrade notes" section before upgrading. This release changes the index format and requires a brief maintenance window to rebuild indexes.

New features

• Trace flame graph view: span-level latency visualization that spots slow nodes at a glance, with click-through drill-down into the logs a Span produced;
• Log search aggregation cache: repeated aggregation queries are cached for 60 seconds, so refreshing the console no longer recomputes the same result;
• Agent canary auto-upgrade: collection Agents can upgrade automatically in batches by ratio, rolling back to the previous version automatically on failure;
• Dashboard variables: dashboards now support variables such as time range and service name, so one template switches across services instead of building a board per service.

The flame graph is the headline item. Instead of scanning a flat list of Spans and mentally summing durations, you now see the request's time budget rendered as nested bars, with the slowest child highlighted. Click a Span and the log panel updates to the exact lines it emitted — which turns "why is this request slow" into a two-click answer.

Performance improvements

• The log write path was reworked, raising single-node throughput from 15,000 to 30,000 lines per second;
• Hot/cold tier query optimization cut cross-tier aggregate query P95 by about 50%;
• The index compression algorithm was upgraded, reducing storage usage by about 30% for the same data volume.

The throughput gain comes from batching writes and reusing buffers on the ingest path, so existing deployments see the improvement without any config change. The storage reduction is the more valuable number for most teams, since it directly lowers the object-storage bill for archived logs.

Bug fixes

• Fixed delayed recovery notifications not firing after an alert's silence period ended;
• Fixed an intermittent Agent high-CPU issue on Kylin V10;
• Fixed inaccurate SQL query results when the time range spans midnight;
• Fixed the trace sampling rate being truncated at decimal places;
• Fixed field order scrambling when exporting audit logs.

Upgrade notes

1. Back up your config files and database before upgrading;
2. This release changes the index format — rebuild the index once after upgrading. The tool is built in: a single jjhub reindex command does it;
3. Agent and server versions must be upgraded together, or the report format may be incompatible;
4. For canary upgrades, upgrade Agents first and the server second, and reverse that order on rollback;
5. After upgrading, run a quick smoke test: write a sample log line, run an aggregation query, and open a trace to confirm the flame graph renders.

The index rebuild runs in the background and can be monitored with jjhub reindex --status. Rebuilding is I/O-heavy, so schedule it outside peak hours and expect a temporary bump in disk usage while the new index is written alongside the old one.

Known issues

• The new flame graph view renders best in Chromium-based browsers; Safari may drop some hover tooltips on very deep traces. Fixed in the next patch release.
• The 60-second aggregation cache means newly ingested logs can take up to a minute to appear in a cached dashboard widget. If you need real-time during an incident, append nocache=1 to the query.
• Agents upgraded via the new canary mechanism won't roll back if the failure happens during the final "commit" step; in that rare case, re-run the upgrade manually.

What's next

v2.7.0 is already in planning. The two items at the top of the list are full OpenTelemetry Logs support (exporting log records directly through OTLP rather than relying on the file-tailing Agent) and a native eBPF-based host profiler for CPU and memory flame graphs without instrumenting the application. If either matters to you, the release notes issue tracker is the place to vote.

A note on the performance numbers: the throughput and compression figures come from our internal benchmark environment (16 cores, 64 GB, write-only load). Your production gains will depend on your log shape and how many fields are indexed — more fields usually compress better but reduce ingest throughput. Run a benchmark against your own typical logs before sizing capacity.