Taming Alert Storms: From 200 Alerts to One Actionable Notification

Why alerts keep crying wolf

At three in the morning a database failover causes a thirty-second hiccup. In those thirty seconds the order service times out, the payment service floods its retry queue, the gateway starts returning 5xx, and Redis connection-pool alerts light up — one root cause cascading into more than two hundred notifications. The on-call engineer is woken by the phone, spends ten minutes digging the first real error out of the flood, then another twenty dismissing everything that does not matter. By the time the incident is actually triaged, the damage is long done.

The problem is not that there are too many alerts. It is that the alerts have no structure. A modern alerting module is not judged by whether it can push a message; it is judged by whether every message it pushes deserves a glance. That is exactly what Observe's alerting pipeline is built around, and it does the heavy lifting in four layers.

Four lines of defense: grouping, inhibition, silence, severity

Grouping merges alerts that share the same dimensions into a single entry carrying a count and first/last timestamps, instead of one line per event. Five hundred 500 errors from the same service become one alert no matter how many times they fire inside the grouping window. Two knobs control the behavior: group_wait holds the first notification while sibling alerts arrive, and group_interval controls how often the engine re-notifies you that the group is still active — so a long outage surfaces as a measured heartbeat, not a firehose.

Inhibition suppresses derived alerts once a root-cause alert is active. When you declare that “database unavailable” is a root cause, downstream alerts such as “API timeout” and “connection pool exhausted” are silenced automatically. Fix the root cause and the symptoms vanish on their own — there is no point paging the same engineer twice for one failure. Inhibition rules are expressed as a directed relationship: the root cause suppresses its dependents, never the reverse, which keeps the graph honest.

Silence covers planned change. Releases, migrations and load tests are expected to trip alerts, so you silence the relevant rules for the duration of the work. A silence must always carry an expiry time; a forgotten silence that never expires is how a real outage goes unnoticed the following week. Observe shows active silences right on the alert page, so a stale one is visible at a glance.

Severity routes notifications by importance rather than treating everything equally. P0 pages by phone plus SMS, P1 goes to WeCom, and P2 stays on the dashboard where it does not interrupt anyone. The channel matches the event's blast radius, so a single flaky endpoint never wakes a human being.

Example: compressing 200 alerts into one

Alert rules are declarative YAML. The rule below aggregates “order-service error rate above 5%” by error code and inhibits the downstream pay-service timeout alert:

alert_rules:
  - name: order-service-error-rate
    expr: error_rate(service='order-service', window='5m') > 0.05
    severity: P1
    group_by: [service, error_code]
    group_wait: 30s
    inhibit: [pay-service-timeout]
    notify: [wecom-oncall]

Add a second, root-cause rule: when the database connection pool is exhausted, the order, payment and gateway alerts are all suppressed by the inhibition graph, and the on-call engineer receives a single “DB connection pool exhausted” message. Here is the before-and-after in concrete terms: before these two rules existed, a database blip produced roughly 200 notifications across four channels and three people were paged; afterwards, the same blip produces one P0 notification to one person, and the rest of the team stays asleep. That is the entire job of an alerting system, and it is config, not code.

Practical recommendations

• Spend half a day mapping the dependency topology of your alerts, separating root causes from symptoms. Inhibition rules are meaningless without a real understanding of which failures cause which.
• Keep grouping dimensions coarse at first: start with group_by: [service] and refine to error_code only once the first layer is stable and quiet. Over-granular groups are just a different kind of spam.
• Review alert data every month. Which rules have not fired in 30 days? Which are silenced most often? Delete or retune them — an alert that never fires is dead weight, and one that always fires is noise.
• Always attach an expiry to every silence, and route P0 sparingly: every page should be a question you are willing to wake someone up to answer.
• Wire alerting into your on-call rotation rather than a fixed group. A converged alert is only useful if the single person it wakes is the person who can actually fix it.

The value of alerting is waking the right person at the right time — not proving that the system is still on fire.