JJHub OBSERVE v2.4.0: Adaptive Sampling and Smarter Alert Noise Reduction

Overview

v2.4.0 focuses on three things: smarter sampling, quieter alerting, and stronger search. Changes are listed by area below; read the compatibility notes at the end before upgrading. The full list of bug fixes is in the release notes in the console.

These changes map to two recurring complaints: traces that should exist but don't because sampling dropped them, and alert channels that are too noisy to trust. If either describes your team, this release is aimed at you. Existing 2.3.x users can upgrade in place without re-instrumenting anything.

Tracing: tail-based sampling

Sampling grows from head-based to tail-based, fixing the high-traffic problem where normal requests get sampled and failing ones get dropped. With head-based sampling, the decision is made when a span starts, before you know whether it will fail - which means your most interesting traces are exactly the ones you lose. Tail-based sampling flips this: decide after the span completes, once you know the outcome.

• Tail-based sampling: keep the decision until the span completes, so traces with errors or timeouts are always retained
• Capture-on-error: spans with status = ERROR or containing an exception are kept at 100%, regardless of base rate
• Per-service sampling rates: 100% for core paths, 1% for ordinary endpoints

Config example:

sampling:
  mode: tail
  base_rate: 0.1
  keep_on_error: true
  per_service:
    trade-core: 1.0
    admin-api: 0.05

Pilot this on a non-core service first and watch storage growth before enabling tail sampling everywhere - keeping every error trace does increase retention, and you want to see the actual number before committing. The base_rate still applies to healthy traffic, so the cost is bounded; only the error tail is always captured.

Alerting and log search

Two new noise-reduction features land in the alerting module:

1. Time-windowed thresholds: set different thresholds per time period for the same metric - for instance, tighten the latency threshold to 200ms during the open session and relax it during the lunch break
2. Flap suppression: one-off jitter enters a pending state and only fires after persisting beyond a configured duration, so momentary spikes stop flooding the channel

These two features are designed to work together: time-windowed thresholds handle the predictable shape of your traffic, while flap suppression catches the unpredictable spikes. In testing with early adopters, combining them cut alert volume roughly in half without dropping a single real incident.

Silence windows can also follow the holiday calendar automatically, muting low-traffic alerts on closed days. The goal is fewer pages, not fewer alerts - the important signals should still get through, and the noise should disappear. If you already tag alerts by business hours, the calendar integration replaces that manual bookkeeping.

Log search now supports subqueries, letting you aggregate then filter in a single statement:

SELECT * FROM logs
WHERE service IN (
  SELECT service FROM logs
  WHERE level = 'ERROR' AND timestamp >= now() - 300
  GROUP BY service
  HAVING count(*) > 10
)

This returns every log line from services that exceeded 10 errors in the last 5 minutes - no second query or manual stitching. A common first use is correlating a spike in one service with the requests it made downstream, in a single view rather than two tabs.

Other improvements and upgrade notes

Other fixes: resolved intermittent OTLP/gRPC connection drops under high concurrency; improved indexing throughput by ~20% on ARM64; added a funnel chart component to dashboards for stage-by-stage conversion rates; fixed missed alert notifications at holiday-calendar boundaries. The ARM64 indexing gain matters specifically for Xinchuang deployments on Kunpeng and Phytium, where every percentage point of throughput translates directly into fewer data nodes.

This is a minor release and stays compatible with v2.3.x configs. Back up the database and index configuration before upgrading, then verify sampling policies and alert rules behave as expected. If you hit issues, open a ticket in the console with your version and the relevant traceId - that context makes a real difference in how fast we can help.