← Back to blog
Changelog 4 min read 炬鲸团队

Observe v2.4: Metric Aggregation Queries, Alert Silence Windows and Trace Sampling

v2.4 adds metric aggregation queries, schedulable alert silence windows and per-service trace sampling, plus search performance work and fixes. Upgrade notes included.

New: metric aggregation queries

Metrics were previously limited to built-in dashboards. If you wanted to slice the data by a dimension that was not on a panel, you had to file a request and wait. v2.4 changes that with PromQL-like aggregation queries — you can filter, aggregate and apply functions directly:

sum(rate(http_requests_total{service="pay-service"}[5m])) by (status)

Common functions — rate, sum, avg, min, max, histogram_quantile — are supported out of the box, along with label filtering and time-range selection. Aggregation results can be wired straight into an alert rule's expr, so custom-metric alerts no longer need a ticket or extra code. The query engine is the same one that powers dashboards, so an expression that works in the editor behaves identically when it fires as an alert — one expression, two uses.

New: alert silence windows

Previously, quieting alerts during maintenance meant pausing rules one by one and then re-enabling them one by one — miss a single one and you are paged at 3 a.m. for a backup you scheduled yourself. Alert rules now support silence windows in two forms:

  • Fixed periods: e.g. daily 02:00–04:00, to cover routine backups
  • Scheduled tasks: linked to your release calendar, auto-silencing during deploy windows

Alerts are still recorded while silenced; only notifications are suppressed. Normal behavior resumes automatically when the window ends, so there is nothing to re-enable and nothing to forget. Silence windows can be layered on top of escalation, and P0 severities can be configured to "never silence", so a critical path is not missed during a quiet period. The most common mistake is silencing too broadly — keep the fixed window narrow, and tie release-related silence to the actual deploy window so you never accidentally mute a real outage that happens to overlap.

New: per-service trace sampling

Trace sampling is no longer all-or-nothing. Set the sampling rate per service — drop high-volume, low-value services (like static-asset proxies) to 1% while keeping core transaction services at 100%. This balances trace coverage against storage cost, which is where most tracing budgets actually leak: you were paying to store a complete trace of every thumbnail request you will never look at. Configure it under Service Management → Sampling Settings; rules match by service name and take effect immediately. A typical setup keeps the checkout and payment services at 100%, the search and catalog services at 10%, and static assets at 1% — no service restart required, because the sampling decision happens in the pipeline, not in the application.

Improvements and fixes

  • Log search performance: average query latency down about 35% for large time ranges, most visible on queries spanning days where the old index strategy forced a full scan
  • Fixed keyword alerts failing to fire when logs were written late
  • Fixed unstable ordering of time fields on Dameng databases causing duplicated pages
  • Reduced Collector memory usage during batched uploads
  • Added secondary sort by field, making pagination stable on large result sets

Upgrade notes

This release changes the metric storage schema, so a data-migration script must run after upgrading; expected duration scales with existing metric volume. Upgrade during a low-traffic window and back up your configuration first — the migration is one-way, and a rollback means restoring from backup. Also upgrade the Collector to 0.9x or later before upgrading, because metrics sent by older Collector versions are not recognized by v2.4 and will be silently dropped.

Known issue: on Dameng, the metric migration script may time out when using an older JDBC driver. Upgrade the Dameng driver to 8.1.2.147 or later before running the migration.

Getting started

  • Metric queries: open Metrics → Query, paste an expression and click Run. Save any result as a dashboard panel from the toolbar, or copy the expression into an alert rule's expr.
  • Silence windows: edit an alert rule, open the Silence tab, and add either a fixed daily window or a link to your release calendar.
  • Sampling: go to Service Management → Sampling Settings and add a per-service rule; it applies to incoming spans immediately.

All three features ship in both the cloud and private deployments. If you run a private deployment, do not skip the migration step above — the new query engine reads the migrated schema and returns empty results until it finishes.