On-call Alerting Workflow
Route alerts, define SLOs and write a runbook.
0 of 6 steps
What you will end up with
SLO-based alerts
A routing and escalation flow
A practical runbook
Build steps
0/6Pick a target, like 99.5% of requests succeeding over 30 days, based on what users actually notice. The SLO is the speed limit sign, chosen because that is roughly the speed traffic actually needs.
Alert on user-facing symptoms, like error rate or latency, instead of causes like high CPU. Alerting on symptoms means the smoke detector goes off for smoke, not for someone lighting a match nearby.
Set up a route tree so low-severity alerts land in Slack and high-severity ones page someone directly. It is a mail sorter deciding between a sticky note and a phone call depending on how urgent it is.
Group related alerts so twenty things firing from one incident become a single notification, not twenty. Grouping bundles a stack of related mail into one envelope instead of twenty separate letters.
For each alert, write down the first three things you would check, before you are the tired person paged at 3am. A runbook is the checklist you wish you had the last time you were paged half asleep.
Deliberately break something in staging and see if the alert fires, routes correctly, and the runbook actually works. It is a fire drill run on purpose, so the real fire is never the first time you test the exits.
Before you start