One database blip can fire 300 alerts and push on-call engineers to mute their phones. This article shows how Observe converges alerts with grouping, inhibition and silence, with YAML examples and escalation advice.
At 2 a.m., the core database hits a 30-second slow query. In that half minute a chain reaction unfolds. The slow query fires a database monitoring alert. The order service that depends on it starts timing out and returning 5xx errors. The gateway's error rate spikes past its threshold. Spans across every trace suddenly exceed their latency budgets and trip their own alerts. By the time the on-call engineer picks up their phone, more than 300 alerts have arrived from five different systems — SMS, email and the IM group all lighting up at once. The first ten minutes are spent muting, not investigating.
An alert's job is to tell you what broke, not to tell you how many times it broke. When one failure generates dozens of alerts, the instinct is not to debug but to silence the phone. That is alert fatigue, and it is the most common way an alerting system fails: a system that never stops ringing doesn't get tuned, it gets ignored. And then, on the day a real outage hits, nobody is watching. The frustrating part is that all 300 alerts are technically correct. The database was slow, the order service did time out, the gateway's error rate really did spike. Each individual alert told the truth; together they told a lie — the lie that there were 300 separate problems to solve.
Observe's alerting engine applies three passes between "fired" and "delivered", in a fixed order: group first, then inhibit, then let silences drop the rest. Each pass removes a different kind of noise.
Grouping merges alerts with similar labels under the same rule into a single notification. Group by service and host, and ten "CPU above threshold" alerts from one machine collapse into one message that carries their combined timeline — instead of ten separate pings burying the one that actually matters. A good grouping key reflects how you'd naturally describe the problem: by service and machine for infrastructure alerts, by topic or queue for middleware alerts.
Inhibition encodes dependencies between alerts. If the database is down, downstream connection failures are inevitable and tell you nothing new. Configure "database down → suppress downstream connection-failure alerts", and those alerts are held back until the database recovers, then released automatically. Inhibition is the pass that stops a single blip from becoming a 300-alert storm in the first place. Without it, grouping can only shrink the storm, never prevent it.
Silence covers planned work — releases, maintenance windows, load tests — so expected noise never reaches anyone at all. The key discipline is to always set an expiry: a silence with no end date quietly becomes a place where real failures hide.
On the Alert Rules page, an inhibition rule looks like this:
inhibit:
- source: "db_mysql_down"
target: "service_connection_failed"
equal: ["env", "cluster"]The equal list scopes the suppression: it only applies when environment and cluster match, so you don't swallow identical alerts in other environments where the failure is real. Grouping is just as direct:
group_by: ["service", "host"]
group_wait: 10s
group_interval: 5mgroup_wait is the merge window — alerts arriving within ten seconds are combined into one. group_interval controls how later alerts append to that same notification rather than starting a new one. Together they turn "one alert per failing check" into "one alert per incident".
Convergence solves "too many alerts"; rotation solves "who responds". Pair a rotation schedule with escalation rules: if a P0 alert goes unacknowledged for ten minutes, the second-line engineer gets a phone call; if it's still unacknowledged after thirty, it pages the manager. Wire the three together — converge first, deliver next, escalate last — and on-call becomes survivable instead of exhausting.
One simple rule to leave with: the alerting system should not be louder than the outage itself. Quiet the noise first, then talk about smart analysis. Most teams that complain about "too many alerts" don't need a smarter alerting system; they need these three passes applied to the one they already have.
A useful way to audit your current setup is to pull one week of delivered alerts and count how many share a single root cause. If the ratio is well above one-to-one, you have a grouping or inhibition gap, and the configuration above is exactly where to start closing it.