Alert storms are the worst part of on-call. This article breaks down the alert engine design: aggregation windows, root-cause grouping, silence windows, four-level severity with auto-escalation, tag routing, and on-call rotation.
The hard part of an alerting system is never "can we send it" — it's "does it actually need to be handled." A service with a million requests a day will page someone constantly if every blip triggers an alert; after two days the on-call person goes numb and real failures drown in the noise. When we rebuilt the alert engine, we set a single goal: cut what on-call has to handle from hundreds of alerts a day to a few dozen, without missing real incidents. This article breaks down how that noise reduction and prioritization work.
The first layer of noise reduction happens before any rule fires: aggregate first, alert second. Raw metrics tick once per second, and comparing a single point against a threshold produces constant jitter. We changed the evaluation target from "a single point" to "an aggregation window" — for example, "alert only if the error rate stays above 1% for five minutes." Short-lived spikes never reach the alerting path. Rules expose two knobs — duration and minimum trigger count — so users tune sensitivity per service.
In practice a rule looks like this:
alert: order-error-rate
expr: rate(order_errors[5m]) / rate(order_total[5m]) > 0.01
for: 5m
labels:
severity: P1
team: order
annotations:
summary: "订单接口错误率超过 1%，持续 5 分钟"The for clause does the heavy lifting: it holds the alert in a pending state until the condition has been true for the entire window, so a 30-second spike never fires. The same expression style applies whether the signal is a metric, a log-derived counter, or a trace error rate — the alert engine treats them uniformly.
The second layer is deduplication and convergence. Once a rule fires, if its state hasn't changed it sends exactly one notification and then stays in a sustained state instead of re-paging. Alerts that share a root cause — say one machine dies and ten health checks on it all fail — collapse into a single parent alert with ten child items, not ten separate pages.
The third layer is the silence window. For planned deploys, load tests, and datacenter cutovers, create a silence window in advance and nothing fires during it. Silence windows filter by rule, data source, or tag, and support both one-off silences and recurring ones, like a weekly 3 a.m. backup job.
Alerts need severity levels. Otherwise a "disk at 75%" notice and a "core API returning 502" alert look equally urgent on a phone. We use four levels: P0 service down, P1 degraded, P2 performance deterioration, P3 informational. Severity is not hard-coded per rule — it is inferred from blast radius. Anything touching a public-facing core endpoint, exceeding a user-impact threshold, or lasting too long is automatically escalated to P0/P1.
Severity also feeds back into deduplication: if a P3 fires and later escalates to P1, the engine closes the P3 and re-notifies at the higher level, so the escalation is visible instead of silent. Severity drives the notification channel directly: P0/P1 go out via phone call, WeCom, and SMS; P2 via WeCom and email; P3 lands in the todo list and stays quiet. On-call staff know at a glance what to look at first.
After prioritization comes routing. Alerts route by tag — service=order goes to the order team, env=prod only to whoever is on shift. The on-call rotation table assigns primary and backup responders per week, and alerts automatically go to the current responder's WeCom and phone. If the primary doesn't acknowledge within 15 minutes, the alert escalates to the backup, then to the team lead on further timeout.
Acknowledgment and escalation must form a closed loop. Every alert supports acknowledge, claim, reassign, and close — all logged. A weekly alert report tracks false-positive rate, mean response time, and escalation count, and the data forces rules to converge: a rule with over 30% false positives for two straight weeks is a rule whose threshold needs redefining.
A good alerting system is ultimately measured by whether on-call staff can sleep at night. The more solid the noise reduction and prioritization, the lower the chance a real failure slips through.