Alerts Without the Noise: Noise Reduction and Escalation in Observe

More alerts is not better

Here is a real scenario. A payments team wired up alerts for "error rate above 1%", "response time over 200ms" and "disk usage over 80%" all at once. The result: a single service hiccup produced dozens of notifications within a minute. The on-call engineer quickly formed a habit — mute the alert group first, then take a look. Once alerts lose trust, the real incident is the one nobody opens.

Observe's alert engine is built around a single idea: every alert should be worth opening. To get there, it works in three layers — rule definition, noise reduction, and escalation. Each layer fixes a distinct way that alerting goes wrong, and skipping any one of them will quietly erode the system you built.

Rules: express "what counts as abnormal" precisely

The first failure mode is vague rules. If your threshold is a gut feeling rather than a measured number, you will either miss real incidents or drown in false positives. Observe supports three kinds of trigger conditions that can be freely combined:

• Threshold: cpu_usage > 80 sustained for 5 minutes
• Error rate: error_rate > 0.5% within 5 minutes
• Keyword: the log stream contains OutOfMemoryError or connection reset

A rule can attach several conditions; hitting any one of them triggers it. More importantly, always add a for clause so an anomaly must persist before alerting. This filters out the spikes caused by deploys and autoscaling:

alert: order_service_error_rate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.005
for: 3m

for: 3m means the error rate must stay above 0.5% for three consecutive minutes before the alert fires. That one line eliminates most of the noise caused by releases and scaling events. If you take nothing else from this article, take the for clause — it is the cheapest reliability improvement available.

Noise reduction: merge fifty alerts into one

The second failure mode is duplication. A single root cause — say, a database outage — can fan out into fifty alerts across every service that depends on that database. On-call engineers drown in repetition and stop reading. Noise reduction relies on three techniques:

1. Dimension aggregation. One alert per service and error type, not one per instance or host. All replicas of order-service failing together produce a single alert.
2. Silence windows. Routine operations like a midnight release or planned maintenance can be silenced manually or on a schedule, so the work itself does not spam the channel.
3. Alert grouping. Alerts that share a root cause are folded into a single group that notifies once; subsequent matching alerts just append to that group without pinging anyone again.

Combined with recovery notifications, only the full "fired to recovered" lifecycle deserves attention — all the redundant repeats in between are dropped. An engineer should be able to answer "what is broken and when did it start" from a single glance, not by scrolling back through a wall of identical messages.

Escalation: who gets it, and how long before it goes up

The third failure mode is unaccountable delivery. A message that nobody acknowledges is as good as no message at all. Observe routes alerts by severity:

• P0 (core path down): phone + SMS + WeCom, auto-escalates to the on-call manager after 5 minutes without acknowledgement
• P1 (business impact): WeCom + email, escalates after 30 minutes unacknowledged
• P2 (warning): WeCom group only, no acknowledgement required

Every channel supports retries and timeout escalation, so a message that nobody saw does not quietly disappear into the void. The key design decision is the escalation timeout: it should be short enough that a sleeping on-call never silently drops a P0, but long enough that a false page does not wake the whole team.

Practical advice

For the first week, only enable P0 and P1, and record P2 silently as an observation. Watch what actually fires. Tune the thresholds and the for windows against reality, not against what you guessed during design. Once the team is satisfied with the signal-to-noise ratio, open up the rest gradually. The value of alerting is never in the count — it is in every single alert being acted upon.