← Back to blog
Changelog 4 min read 炬鲸团队

OBSERVE v2.6 Release Notes: Trace Storage Overhaul and Alerting Upgrades

v2.6 rebuilds the trace storage engine for 3x faster cross-service queries, adds silence windows and alert aggregation, and ships Dameng DM8 and ARM support. Includes upgrade steps and breaking changes.

Overview

OBSERVE v2.6 is out. This release does three things: it rebuilds the trace storage engine, upgrades alerting, and rounds out domestic-stack support. Read the breaking changes before you upgrade — this release touches the storage format, so you must migrate data first. The headline number is a 3x speedup on cross-service trace queries, which came from replacing the service-sharded span store rather than from any single query optimization. If you run OBSERVE at scale, this is the release you upgrade for.

New features

Trace storage overhaul. The old span store sharded by service, so cross-service queries required multiple scans. v2.6 switches to hash sharding by trace_id plus columnar storage. Reading a whole trace goes from several random reads to a single sequential scan, dropping cross-service query P99 latency from 2.4s to 0.8s. The new layout also compresses better: span payloads with repeated attribute names roughly halve their on-disk footprint.

Alert silence and aggregation. A new silence window means planned releases and load tests no longer spam the channel. Multiple alerts from the same source collapse into one, annotated with a trigger count, so alert storms stay under control.

Domestic-stack support. New Dameng DM8 database adapter and aarch64 (Kunpeng/Phytium) packages; Kylin V10 installs directly.

Metric pre-aggregation. New Recording Rules pre-compute high-frequency queries like rate() into one-minute granularity results, cutting large dashboard load times in half.

Two smaller additions round out the release. First, the trace detail view now shows a per-span breakdown of time spent in the queue versus in processing, which is where most "why is this slow" investigations actually end. Second, alert rules gained a group_by field so you can route alerts to different channels per service without duplicating rules.

Upgrade steps

Upgrading from v2.5 is a two-step job — migrate the trace data first:

# 1. Back up config
cp -r /opt/observe/config /opt/observe/config.bak
# 2. Upgrade the binary
rpm -Uvh observe-2.6.0.aarch64.rpm
# 3. Run the storage migration
observe migrate --from 2.5 --to 2.6
# 4. Restart and verify
systemctl restart observe && curl -s http://localhost:8080/health

The migration runs asynchronously in the background. For large clusters, run it during off-peak hours — query service stays up throughout. Watch the migrate process output for a completion line before you trust the new format; it prints progress per shard. Before you start, snapshot the trace volume with observe stats --trace to get the total span count, which the migration uses to estimate remaining time — on a cluster with a billion spans this took just under two hours in our testing.

Breaking changes

  • Trace storage format changed. Starting without running observe migrate first makes old data unreadable.
  • The for field in alert rules changed meaning: for: 5m used to mean "fire after 5 minutes"; it now means "fire once the condition has held continuously for 5 minutes". Review your existing rules.
  • The PromQL compatibility layer removed several deprecated functions; see the migration checklist in the docs.

Most teams are unaffected by the breaking changes — the for semantics is the only one that silently changes behavior, and it makes alerts strictly more predictable. Still, grep your alert rules for for: before flipping the switch.

Rollback

If the migration fails mid-way, stop OBSERVE and restore the config backup, then restart with the previous binary. The migration is idempotent per shard, so re-running observe migrate resumes where it left off instead of starting over. Do not run two migrate processes at once — the second refuses to start and logs a lock error.

Known issues

  • On Kylin V10 SP2, the filelog receiver occasionally drops the tail of a log file. A fix is ready for the next patch; the temporary workaround is to point the log path at directory monitoring instead.

What's next

v2.7 is already scoped around two themes: a streaming trace tail that removes the refresh gap during live incidents, and first-class support for the Phytium + UOS combination that several government customers have requested. If you rely on either, keep an eye on the release candidates. As always, this release is backward-compatible with the 2.x API, so dashboards and saved queries keep working unchanged after the upgrade — the only thing that needs your attention is the storage migration above.