Alert storms are the loudest noise on call. This post breaks down Jujing's three-layer noise reduction — dedup, grouping, and inhibition/escalation — with a ready-to-use config and a rollout order that cuts notification volume for real.
A single slow database query can trigger five alerts at once: high CPU, slow SQL, too many connections, downstream timeouts, and caller 5xx errors. All five describe the same incident, yet the on-call engineer gets pinged five times and has to acknowledge each one. Worse, in most teams' alert history, around eighty percent are false positives caused by threshold jitter — CPU spikes to 85% during peak traffic and drops back after thirty seconds, so a fixed 80% threshold keeps firing over and over.
The value of alerting is not volume; it's that every alert is worth acting on. Noise reduction has to solve exactly three problems: duplication, jitter, and cascading.
Jujing splits noise reduction into three layers, applied in order.
Layer one is deduplication. We build a fingerprint from metric name + label set + rule ID. Alerts sharing a fingerprint are sent only once within a window; later arrivals just increment a counter, and the notification shows how many times it has fired.
Layer two is grouping. Alerts arriving in the same window with heavily overlapping labels are merged into a single notification listing the affected hosts or services. Twenty machines hitting a CPU threshold at once become one alert — "20 nodes over CPU threshold" — not twenty.
Layer three is inhibition and escalation. You declare dependencies: when failure A necessarily causes alert B, only A fires and B is silenced. In parallel, alerts get severity levels, and a P1 alert left unacknowledged for ten minutes is escalated to P0 with a phone call.
alert_noise:
dedup:
window: 5m
fingerprint: [metric, labels, rule_id]
group:
window: 10m
by: [service, region]
max_events: 50
inhibition:
- source: node_down
target: service_unavailable
equal: [host]
escalation:
- level: P1
wait: 10m
to: P0
notify: [sms, phone]With this config, the same alert fires once every five minutes; alerts are grouped by service and region within a ten-minute window; a node-down event silences every "service unavailable" alert on that host; and an unattended P1 alert escalates to a phone call after ten minutes.
Start with dedup and grouping — they are the fastest wins with the lowest risk, and they typically cut notification volume in half within a couple of days. Inhibition requires mapping dependencies, so begin with the single most obvious rule: "node down → service unavailable." Add escalation last, piloting with P1 alerts rather than letting P2 pages wake people up.
Jitter is solved by dynamic baselines, not by hand-tuning thresholds. Turn on anomaly detection for key metrics: learn a baseline from thirty days of history and alert only on deviations beyond three standard deviations. That yields an order of magnitude fewer false positives than fixed thresholds. After launch, review unanswered alerts weekly and delete rules that never lead to action — that trims more noise than any feature ever will.