Too many alerts are as dangerous as none. This article shows how Observe collapses thousands of daily alerts into a few dozen actionable events through grouping, deduplication, severity routing, on-call rotation and silence windows, with working configuration examples and rollout advice.
Operations teams rarely fear having no alerts; they fear having too many. A single database node flapping can trip four rules at once — connection pool saturation, slow queries, disk I/O and service availability — and within seconds the screen fills with dozens of messages. The on-call engineer cannot tell which one is the root cause and does not dare silence any of them. Over time the alerts become background noise, and the incident that actually matters gets buried in the scroll.
This is not an exaggeration. Across a sample of customers we found that a mid-size team typically produces 2,000 to 5,000 raw alerts a day, but fewer than 30 of them actually require a human. The other 99% are duplicates, flapping and low-value reminders whose only effect is to wear down the on-call engineer until they mute their phone at 3 a.m.
Observe's alert engine focuses on two jobs: converging thousands of raw alerts into a handful of actionable events, and routing each event to the person who should act on it. Convergence comes from grouping and deduplication; delivery comes from severity routing.
Grouping clusters alerts automatically by label dimensions. Alerts triggered by the same service, host and metric within one time window are merged into a single aggregated event that records the trigger count and peak value, instead of being pushed one by one. The configuration is short:
group_by: [service, host, alertname]
group_wait: 30s
group_interval: 5m
repeat_interval: 4hEach parameter controls a distinct stage:
group_wait holds the first notification for 30 seconds so one wave of flapping collapses into a single eventgroup_interval caps how often new alerts inside the same group remind you, here every 5 minutesrepeat_interval re-notifies only every 4 hours for alerts that have not recovered, so the phone does not buzz all nightDeduplication solves the problem of "the same alert firing repeatedly"; grouping solves the problem of "a batch of alerts that are really one incident". Only the two together make a difference. A network partition might make 40 hosts report "heartbeat lost" at once — after deduplication that is still 40 alerts; grouped by failure domain, they become a single "network anomaly in room X" event, and the on-call engineer immediately knows who to call.
How well this works depends on label design. If labels are too coarse — grouping only by service, say — failures on different hosts get incorrectly merged; too fine, and nothing converges. Start with service plus host, run for a week, then add labels based on which alerts always fire together.
Alerts are classified P0 through P3 and mapped to different channels and recipients:
The basis for severity is not "how important you feel it is" but "who must know, and how fast, when it breaks". A useful test: if this alert paged you at midnight, would you get up to handle it? If not, downgrade it or make it record-only.
The other half of routing is the on-call schedule. Alerts should reach whoever is on shift, not everyone on the team. Observe supports rotating schedules with automatic handoff, so an alert never wakes up the colleague who already signed off. In multi-team setups you can also route by service: payment alerts go to the payment team's on-call channel, order alerts to the order team, and neither sees the other's noise.
During release windows, load tests or known incidents, mute the relevant alerts so they do not mislead anyone. Silence windows can be scoped by service, host or time range and expire automatically. A routine 2 a.m. backup, for example, can silence its disk-I/O alerts for 30 minutes instead of paging someone every night about an alert that was never a problem.
One detail that is easy to overlook: silence windows should tie into change management. Open the silence automatically when a release starts and close it when the release ends, rather than relying on someone to remember. The biggest risk of manual silences is forgetting to turn them off, which then silences a real incident later.