A walkthrough of Jujing OBSERVE's alerting engine: threshold, dynamic-baseline, and composite rules; aggregation and deduplication; escalation and on-call scheduling. Follows one CPU alert through the full pipeline to raise signal-to-noise.
The goal of an alerting system is not to surface every anomaly, but to make sure the person on call receives only the alerts they actually need to act on, at the right time. Most systems do the opposite: a single database hiccup cascades into dozens of alerts, and the root cause gets buried in the noise. Jujing OBSERVE's alerting engine attacks the problem at three points: how rules are written, how alerts are deduplicated, and how notifications are delivered. Before touching any of it, measure where you stand — pull the last month of alerts and count how many actually led to a change. Most teams find that number is depressingly low, and it's the only baseline that tells you whether tuning is working.
The simplest is the threshold rule, such as cpu_usage > 90% sustained for 5 minutes. It's direct but has two flaws: a fixed threshold false-alarms constantly during low-traffic periods and stays silent during slow-leak failures. So Jujing offers a dynamic baseline: the system fits a band around each metric from the last 7 days of history and fires only when the actual value deviates beyond a configured factor. It catches off-peak anomalies and holiday traffic-shape changes that fixed thresholds miss.
Two things to know about the baseline before relying on it. First, it needs a warm-up — a full week of history at minimum, ideally two, because it models weekly seasonality. Second, sensitivity trades recall for precision: start with medium, which flags roughly a 3x deviation from the band, run it for a week, and only then tune. Cranking sensitivity to high on day one is how you drown your on-call channel.
The third type is the composite expression, which combines conditions with logic:
avg(cpu_usage) > 80 and (error_rate > 0.05 or queue_depth > 1000)These rules suit cases where several symptoms must co-occur before intervention is warranted, cutting false alarms from single-metric jitter. One practical rule when writing rules: ask "what concrete action can the on-call engineer take from this alert?" If there's no answer, don't create the rule.
Alert storms are the on-call team's worst nightmare. Jujing applies three layers of noise reduction. Deduplication: an identical alert is sent only once within a cooldown window; later triggers just increment a counter instead of re-paging. Grouping: multiple alerts from the same cluster or service merge into one notification with a list of affected instances, so "service X is down" arrives once rather than once per pod. Silence: temporarily suppress alerts by time window (release windows) or by label (env=staging), with scheduled activation and manual release.
The first two layers are on by default; silence rules are configured under Alert Center → Silence Management. In practice, this cuts on-call channel volume by more than half. Use silence rules deliberately, though — a silence that outlives a release window is how alerts quietly die.
After noise reduction comes "deliver to the right person." Jujing supports escalation: if an alert isn't acknowledged within N minutes, it's automatically promoted — first to the on-call engineer, then to the team lead after 10 unacknowledged minutes, then to the tech lead after 30, so alerts never just sit unread. It also ships on-call scheduling: set a weekly rotation once, and alerts route to whoever is currently on call without manually editing notification targets.
Notification channels cover email, SMS, WeCom/DingTalk/Feishu, and webhook — the webhook can push alerts into your own ticketing system or IM bot. Best practice: assign an explicit severity (P1/P2/P3) to every rule and route escalation chains and channels by severity — P1 goes to phone plus SMS, P3 only to group chat.
Take "order-service CPU above 90%": the dynamic baseline detects the deviation, an alert is created, grouped with other alerts from the same service, enters the escalation chain as P1, routes to this week's on-call engineer, who acknowledges in the console and jumps to the linked trace and logs. Every step is recorded, so afterwards you can look up the alert and pull the metric snapshot, related traces, and logs for a grounded postmortem. If you want to track whether the whole thing is healthy, watch two numbers: the share of alerts acknowledged within your SLO, and the share that lead to a change. When the first is high and the second is low, your rules are still too noisy.