Too many, too granular, no context — the universal on-call complaint. This post walks through OBSERVE's four alerting design pillars: multi-dimensional aggregation and dedup, silencing and inhibition, alerts that carry trace/log evidence, and escalation tied to schedules.
What Alert Fatigue Really Is
Alert fatigue isn't simply "too many alerts." It's the disconnect between an alert and actionable information: a "CPU over 80%" alert doesn't tell you which business is affected or what to do next. The on-call engineer's first reaction becomes "this one again," and apathy sets in.
OBSERVE's alerting design goal fits in one sentence: every alert either auto-recovers or jumps straight to the root cause. Here are the four pillars.
1. Multi-Dimensional Aggregation and Dedup
One fault fires dozens of alerts. We aggregate by fault dimension, not by alert rule:
- Aggregation dimensions are configurable:
cluster + service, availability zone + app, and so on. - Events within the same dimension and window merge into one alert that shows blast radius (how many instances, how many services) instead of a long list.
- The default window is 5 minutes; new events in the window only bump the counter, not open a new alert.
The result: one network blip in a datacenter collapses from 200 alerts into 3 aggregated ones, each stating its blast radius.
2. Silence and Inhibition
- Silence: for planned changes and known maintenance windows, mute manually or on a schedule so known issues stop flooding.
- Inhibition: topology-based suppression. A node going down necessarily triggers every service on it; one "host down inhibits all its service alerts" rule leaves only the root-cause alert.
Inhibition is the single most effective noise reducer, but also the costliest to configure — start with the host → container → service hierarchy and layer it up.
3. Alerts That Carry Evidence
This is the key step away from fatigue: the alert carries its own evidence links.
Every alert detail automatically includes:
- Trace query — a link to the slow/error trace top list for that service in that time window.
- Log query — a pre-built log search for
service + level=ERROR + time window. - Change correlation — deployments and config changes for that service in the last 30 minutes.
When on-call opens an alert, the first screen shows "what changed + where the slow traces are + where the errors are," not a lone chart.
4. Escalation and On-Call Schedules
- Multi-level escalation: an unacknowledged alert escalates after N minutes; different severities take different chains.
- On-call routing: alerts route to the current responder, with rotation, timezone, and holiday coverage.
- Ack as record: who acknowledged, when, and how it was handled is all logged, forming an auditable on-call history.
Adoption Tips
- Get severity right first: P1 is life-threatening, P2 needs handling, P3 is informational. Don't mark every rule P1.
- Before publishing a rule, ask: "Does the person receiving this alert know what to do next?" If not, add an evidence link before adding the alert.
- Review silenced and ignored alerts weekly — those are the best candidates to optimize or delete.
Make "actionable" the first acceptance criterion for alerts, and fatigue recedes as the evidence chain fills in.