The Alerting Engine and On-Call Management: Rules, Noise Reduction, Silences, and Escalation

Three ways to write an alert rule

OBSERVE's alert rules support three styles, so different teams can keep the habits they already have instead of being forced into one notation:

• PromQL: for teams already on Prometheus, existing rules port over directly — rate(http_requests_total{status="500"}[5m]) > 0.01 works as-is, along with histogram quantiles, rate windows, and label selectors. If your rule library already lives in Prometheus, this is the path of least resistance;
• Threshold expressions: a plain form for people who don't write PromQL. Pick a metric, an aggregation (avg, max, p99), and a threshold, and the UI generates the rule. It covers the common 80% without anyone learning a query language;
• Log alerts: triggered straight from a log query — for example, "more than 100 lines with path=/api/pay and level=ERROR in the last 5 minutes." This turns logs into an alert source without first converting them to metrics, which matters for error signatures that don't have a clean metric.

All three converge on one evaluation engine, so the evaluation interval, for duration, and label-matching behave identically. You don't end up running several rule systems that each have their own quirks and their own way of sending a page.

Noise reduction: don't drown the on-call engineer

An alert's value depends on its signal-to-noise ratio, and most alert fatigue comes from noise, not from too few alerts. OBSERVE applies four layers of reduction, in order:

1. Grouping: alerts from the same rule and label set merge into one, with a counter incrementing instead of a wall of identical messages;
2. Dedup window: an identical alert only notifies once within the window, so a metric flapping around the threshold doesn't fire every evaluation cycle;
3. Recovery delay: after a metric returns to normal, the "recovered" message waits three minutes by default, so a brief dip back into the red zone doesn't page you twice for the same incident;
4. Inhibition: when a node goes down, every service alert on that node is suppressed by the "host unreachable" alert. Without inhibition, one dead host becomes hundreds of derivative alerts that bury the actual root cause.

Each layer is configurable in the UI and takes effect immediately — no restart, no config-file edit. The goal is that a real incident produces one clear page, not a flood.

Silences and maintenance windows

Deployments and planned changes need their alerts suppressed ahead of time, and doing this right means separating two concepts people often conflate:

• Silence: a temporary mask matched by label — silence everything with env=staging until tomorrow morning, or mute one noisy service while you investigate a known issue. It's ad-hoc and label-driven;
• Maintenance window: bound to specific services or hosts, with a start and end time. Alerts are still recorded during the window but not sent, and the window ends automatically. Alerts inside a window get tagged, so a postmortem can tell a real failure from planned churn.

Both support schedules — the weekly Thursday-night release, the monthly database failover drill — so you set them once instead of remembering to toggle them by hand. The tagged maintenance alerts are the part teams most often overlook: without the tag, a week later you can't tell whether that spike was a real incident or just the deploy you forgot about.

On-call and escalation

Notification channels cover WeCom, DingTalk, Feishu, email, and Webhook, so alerts land where your team already looks. The on-call schedule rotates weekly and supports multi-level escalation:

• The alert first goes to the current on-call;
• If nobody acknowledges within five minutes, it escalates to the backup, then to the team lead;
• Every acknowledge, reassign, and close is timestamped, leaving an auditable on-call trail for review.

Permissions are separate for alert rules, schedules, and silences: the on-call engineer can only acknowledge and close alerts, not edit rules — which stops the "quietly raise the threshold" reflex that silently erodes an alerting system's credibility. For teams running multi-site or dual-active setups, alert routing can also be scoped by region, so a failure in one data center pages the people responsible for that data center.

Where to start

Alert fatigue is the most common failure mode, and the fix is usually subtraction, not addition. Start by alerting only on metrics that actually affect the business, and keep P1 rules to a single screen. For the first week, review the false-positive rate daily and delete — or downgrade to a report — any alert nobody ever responds to. Write new rules in staging and let them run for a week to confirm no false negatives before they reach production. A good alerting setup is one where every page is actionable; if you can't name the action, the alert probably shouldn't exist.