Alert tuning
Skill SkillMedev/incident-response-command/skills/alert-tuning
SRE skills for triage, runbooks, alerting, and SLO-driven reliability ops
npx -y skills add SkillMedev/incident-response-command --skill alert-tuningAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Reduces alert noise by making alerts actionable, symptom-based, and tied to SLOs. Identifies and kills flappy, duplicate, and cause-based alerts. Use when alert fatigue is degrading on-call response quality.
SKILL.md
3.0 KB, as published. Nobody here has run it
Alert Tuning
Every alert that fires without requiring human action trains engineers to ignore alerts. Alert fatigue kills real incidents. The goal: every page wakes someone up for a reason they cannot automate away.
The Audit - Run First
Before writing new alert rules, audit what you have.
- Pull the last 30 days of alert history. Calculate: total pages, pages outside business hours, pages that resulted in no action taken.
- Identify flappy alerts: any alert that fires and resolves more than 3 times in a 24-hour window without a corresponding incident.
- Identify cause-based alerts: anything that fires on a system-internal metric (queue depth, pod restart count, CPU) without a direct user-impact correlation.
- Calculate the actionability rate: (pages with a documented remediation / total pages). Below 80% means your alert set is broken.
Symptom vs Cause
Page on symptoms, not causes. Symptoms are what users experience. Causes are what engineers investigate.
- Wrong: alert on CPU above 80%.
- Right: alert on p99 latency above SLO threshold.
- Wrong: alert on pod restart count above 2.
- Right: alert on error rate above 1% for 5 minutes.
For every cause-based alert, ask: 'Would a user notice if this fired and nobody acted?' If no, delete it. If yes, rewrite it as the symptom the user would notice.
SLO-Based Alerting
Tie alerts to error budget burn rate, not raw thresholds.
- Fast burn: burn rate above 14x for 5 minutes (consumes 1 hour of 30-day budget). Page immediately.
- Slow burn: burn rate above 1x for 1 hour. Ticket or warn.
- No burn: no page, regardless of cause-based signals.
This approach eliminates most threshold-tuning debates and makes alert severity self-documenting.
Fixing Flappy Alerts
- Add a 'for' duration to the alert rule. A threshold must be sustained for at least 2-5 minutes before paging.
- Add hysteresis: alert fires at 95% threshold, resolves only when it drops below 85%.
- For bursty metrics, use a rate or average over a longer window (10-15 minutes) instead of instantaneous value.
Routing and Severity
- Route by SLO impact, not by which team owns the metric. The team that can fix it fastest gets the page.
- SEV1/2-level alerts: immediate page.
- SEV3-level alerts: ticket, notify via Slack, no wake-up.
- Never route the same alert to more than one person simultaneously. Shared ownership is no ownership.
Do / Don't
- Do: delete alerts you cannot improve rather than leaving them muted. A muted alert is a lie.
- Do: review alert history quarterly. Thresholds drift as systems scale.
- Don't: add a new alert without deleting or merging one. Alert sets grow monotonically unless you enforce a budget.
- Don't: alert on absence of data without accounting for scrape intervals. Missing data is not always an outage.