Why this matters
A pager is a scarce channel. Every page interrupts sleep, a meal or a conversation. When most pages need no action, people learn to swipe first and read later. That delay shows up on the night a real outage starts.
Alert noise also hides the alerts that matter. If a pile of pages arrives for a disk filling up and one for checkout failing, the checkout page can get lost in the scroll. How to cut alert noise so on-call can sleep is mostly about removing pages that never change what someone does.
The goal is not zero pages. A quiet pager that misses a dead database is worse than a noisy one. The goal is a pager where each page has an owner, a runbook and a reason to wake someone.
Before you start
Pull the last 30 days of pages from your alerting tool and your on-call schedule. You want a raw list: timestamp, alert name, service, severity, who was paged, and what happened next. If your tool cannot export that, start by enabling a log or a webhook that records every page.
Agree on two definitions before you change anything. A page is something that wakes a person and expects a response now. A ticket is work for the next business day. Write both down and put them where the team can see them.
Check how alerts are routed today: services, escalation policies, quiet hours and any existing silences. Find the person who owns the pager rotation and ask what they ignore. Their list of ignored alerts is your first cut.
How to do it
Work from evidence, not from the alert list. A page deserves to exist if someone acted on it, or if someone would have acted if they had seen it sooner. Everything else is a candidate for a ticket, a dashboard or deletion.
Change in small batches. Remove one noisy group, watch for a week, then remove the next. Big-bang cleanups often remove a detection you needed and teach the team to distrust the cleanup.
- Define the page test. Write a three-line test: urgent, actionable, owned. If an alert fails any line, it does not page. Put the test in the alert repository or the on-call handbook so new alerts have to pass it.
- Tag the last 30 days of pages. Open the page history and tag each page with one of four labels: action taken, no action needed, duplicate, or missed detection. This gives you a ranked list of what to fix first.
- Delete or downgrade no-action alerts. For each alert tagged no action needed, either delete it, move it to a ticket queue, or put it on a dashboard. Do not keep it as a page just in case. That habit is what wakes people up.
- Group related alerts. Use grouping labels so one incident sends one page, not twenty. In Alertmanager, set group_by on service and cluster, and tune group_wait, group_interval and repeat_interval. Add inhibition rules so a high-severity page suppresses the low-severity alerts it causes.
- Alert on symptoms, not causes. Page on what users feel: error rate, latency, failed synthetic checks, queue age. Keep CPU, memory and disk as tickets or dashboards unless they are about to take down a service. A disk alert about capacity is planning work, not a 3 a.m. wake-up.
- Add severity and routing. Give every alert a severity label and a service owner. Route page-worthy alerts to the on-call rotation for that service. Route warnings to a team channel during business hours. Do not send everything to one firehose channel.
- Attach a runbook and a dashboard link. Every page should carry a short runbook: what this means, first three checks, who to escalate to. Link the dashboard and the log query too. If you cannot write the runbook, the alert may not be ready to page.
- Hold a weekly page review. Spend 15 minutes on the top five pages from the past week. Ask what action was taken. If the answer is none, change or remove the alert that week. Track pages per shift and set a simple page budget for the rotation.
How to check it worked
Compare the same two measures before and after: pages per shift and the share of pages that led to an action. A spreadsheet from the export is enough. The direction should be fewer pages and a higher action share.
Ask the people who carried the pager. A short question after each shift works better than a survey: which page could have waited until morning? Write the answers down and fix the top one each week.
Check for missed detection. Look at incidents that customers reported before the pager fired. If the cleanup removed a useful signal, add a symptom-based alert for that failure, not the old cause-based one. A quieter pager is only a win if the important pages still arrive.
Mistakes to avoid
Do not tune thresholds by guess. Raising a CPU threshold just moves the noise. Replace the cause alert with a symptom alert or move it to a ticket.
Do not leave permanent silences. A silence with no end date becomes a blind spot. Put an owner and an expiry on every silence, and review them in the weekly page review.
Do not clean up only after a bad night. Noise comes back as services change. Make the page test part of code review for alert rules, and review the top pages every week. If you need help building the alerting and routing layer, Delivery platform setup is one option: a fixed price of $14,500 over four weeks, with monitoring, alerting and dashboards for the services in scope. Other options include doing it in-house or hiring a specialist.
Frequently asked questions
How many alerts per on-call shift is too many?
There is no universal number. Use the action test: if most pages need no action, the rotation has too many. Set a page budget per shift and review the top noisy alerts every week.
Should CPU and memory alerts page someone?
Usually no. Page on user-facing symptoms like errors, latency and failed checks. Keep CPU and memory as tickets or dashboard signals, unless they predict an outage within minutes.
What is alert fatigue?
Alert fatigue is what happens when repeated low-value pages teach people to ignore the pager. The fix is to remove no-action alerts, group duplicates and route by severity. It is a process change, not a tool setting.