Jujing OBSERVE v2.4: Alert Silence Windows and Tail-Based Sampling

v2.4 ships three major updates: alert silence windows with inhibition rules, enhanced trace sampling, and log search performance improvements. This release responds to two of the most common requests we heard — alerts are too noisy during release windows, and long-tail error traces keep getting dropped by sampling. Details below.

Alert Silence Windows

Previously, every release window meant manually disabling a batch of alerts and re-enabling them afterward — and forgetting one or the other was routine. v2.4 adds silence windows, which let you mute alerts by time range and by label in bulk:

silence:
  - name: "weekly release window"
    schedule: "0 2 * * 4"     # every Thursday at 02:00
    duration: 2h
    matchers:
      - env: "staging"
      - severity: "warning"

Silence windows support cron-style schedules and label matchers, so you can cover "every Thursday release" or "any alert tagged env=staging" without touching individual alert rules. They're scoped and audit-logged, so a mute is itself a visible, reviewable action rather than a silent hand-edit on a rule. Combined with inhibition rules, alerts that share a root cause now fire only once while downstream alerts are suppressed automatically. The on-call dashboard finally stays clean, and the release runbook loses its "remember to re-enable alerts" checklist item.

Trace Sampling Enhancements

• Tail-based sampling. Head sampling misses the long-tail requests that are slow but don't error until the end. v2.4 supports tail sampling, deciding after a span completes based on latency and error — so slow or failing long-tail traces are never dropped.
• Rule-based sampling. Configure per-service or per-API-path sampling rates: 100% for critical endpoints, 1% for edge traffic — balancing coverage against cost.
• Sampling decision visibility. The trace detail page now shows whether each hop was sampled and which rule kept it, ending the mystery of "why is this trace missing?"

Rule-based sampling is configured per service or path, so you can write a rule like "keep 100% of /api/pay traces, keep 1% of everything else" without changing any application code — the platform applies the policy at ingest. Tail sampling, by contrast, needs a decision policy evaluated after each span finishes, which you configure once in the platform:

sampling:
  policy:
    - rule: "always_keep_errors"
      when: "span.status == ERROR"
      rate: 1.0
    - rule: "slow_requests"
      when: "span.duration_ms > 1000"
      rate: 1.0
    - rule: "default"
      rate: 0.1

The first two rules keep every errored or slow span regardless of the default rate — that's what "long-tail errors are never dropped" means in practice.

Log Search Performance

High-frequency GROUP BY queries are roughly 40% faster. The changes include a pre-aggregation cache for large time-range aggregations, inverted-index sharding for high-cardinality fields like uri and user_id, and automatic hot/cold query routing — hot data served from memory, cold data from object storage, invisible to the user. If your searches have felt slow, rebuild the index once after upgrading so the optimizations take effect; high-frequency dashboards open noticeably faster as a result.

Upgrade Notes

Back up your metadata before upgrading, then run:

./bin/observe migrate --from=2.3.x --to=2.4.0
./bin/observe start

The migration is backward compatible with 2.3.x data, and we recommend testing on a staging environment before touching production. Note that enabling tail sampling increases write volume; roll it out on a single service for a week before scaling up. Two reminders from the field: silence windows and inhibition rules are evaluated in that order, so a suppressed alert stays suppressed even if its silence window expires mid-incident — that's intended, and it's why we suggest a short post-release review of what got muted. And if you run multiple tenants, the new sampling policies are tenant-scoped by default, so check the scope before applying a global rule. See the official Release Notes for the full changelog, including minor bug fixes and dependency updates.