A single service blip can fire dozens of duplicate alerts and blow up the on-call phone. This post breaks down the Observe alert engine's grouping, inhibition, silencing, and on-call escalation, with config examples and a way to measure alert health.
A single service blip and forty instances error out at once. Every error fires its own alert, and the on-call phone rings for ten minutes straight. By the time you scroll to the one OOM alert that actually matters, the incident has already spread. There's a worse ending, too: because the alerts are always so noisy, the on-call engineer muted the channel weeks ago, and the one failure that needed a human gets buried in the noise. The value of alerting isn't volume—it's "ring when it matters, stay quiet when it doesn't." Get that wrong and alerts stop being a safety net and turn into noise, and the whole system becomes decorative. So the first step in fixing alerting isn't adding more rules; it's crushing the noise you already have.
The Observe alert engine groups alerts by dimension. This rule keeps error-rate alerts for the same service and cluster to a single notification inside a 30-second window:
alert:
name: service-error-rate
expr: error_rate{service="order-service"} > 5
for: 1m
group_by: [service, cluster]
group_wait: 30s
group_interval: 5mThe notification arrives as "42 alerts grouped," with the details behind a click. The three parameters each own a stage: for is a debounce so a transient spike that recovers on its own never reaches a human; group_wait gathers alerts that land in the same burst before sending the first notification; and group_interval controls how often a grouped alert re-notifies, so the same batch doesn't poke you every minute.
Choosing the right dimensions is the whole game: too coarse and unrelated alerts get mixed together, too fine and you haven't grouped anything at all. Start with service + cluster, then split by environment or region. A classic anti-pattern is grouping by trace_id—every trace is unique, so the grouping does nothing.
Two techniques that don't get enough attention.
The first is inhibition. When the connection pool exhausts, hundreds of downstream "connection timeout" alerts blow up at once. An inhibition rule lets "database unavailable" suppress "connection timeout," so the person on call sees the root cause instead of a wall of symptoms:
inhibit:
source: db_unavailable
target: downstream_connection_timeout
equal: [service]The equal list scopes the rule: suppression only applies when source and target share the listed label values, so a database outage in cluster A doesn't hide a genuinely separate connection problem in cluster B. Inhibition chains are allowed—"network partition" suppresses "database unavailable," which suppresses "connection timeout"—so a dozen alarms collapse down to one root cause.
The second is silencing. Planned releases, load tests, and datacenter cutovers should silence related alerts ahead of time so false positives don't eat your attention. Always set an expiry—forget to, and a real failure slips past while the channel is quiet. A common pattern is to tie silences to a deployment window: silence the "service restart" alert from 2 a.m. to 3 a.m., then let normal thresholds take over.
Before you group anything, give every rule a severity: P1 means a core path is down and someone acts now; P2 is a local anomaly to handle within half an hour; P3 is a latent issue that can wait until today. Severity decides two things—which channel the notification uses, and whether anyone gets woken up at night.
Alerts eventually land on a person. Observe ships with a rotation schedule: weekly rotation with a primary and a backup, ad-hoc shift swaps, and delivery over WeCom Work, email, and SMS—with configurable quiet hours so a low-severity P3 at 3 a.m. waits until morning. The escalation chain runs three stages by default: first-line is notified, second-line is pulled in after five minutes without acknowledgment, and a phone call is the last resort ten minutes after that. Every notification and acknowledgment is logged, so a postmortem can reconstruct who saw what and when, and where the handoff stalled.
A concrete yardstick to close with: track three numbers weekly—false-positive rate (the share of notifications where nobody actually had to act), mean time to acknowledge (MTTA), and mean time to resolve (MTTR). If the false-positive rate stays above 20%, the thresholds or rules are wrong; fix the rules instead of adding headcount. Run new rules in "silent observation" mode for a week first—logging without notifying—and switch on real notifications only once the false-positive rate is acceptable. The biggest enemy of an alerting system isn't a missed alert; it's training everyone to treat the alarm as the boy who cried wolf.