Metric alerts

Actionable Metric Alerts

Instructions

An alert that cannot be acted on at 3 AM is noise. Design alerts as symptoms, attach diagnostics, and keep the total paging volume under a human budget.

1. Alert on Symptoms, Not Causes

Page on what the user feels:

• Crash-free sessions dropped.
• Cold start p95 doubled.
• Checkout error rate spiked.
• Sign-in success rate dropped.

Do not page on causes unless the cause is not visible elsewhere:

• High CPU on a single device -- not a page.
• A new error class appearing in Sentry -- not a page, create an issue.
• A backend 5xx rate spike already paged by the backend on-call -- do not double-page.

2. Multi-Window Burn-Rate Alerts

For every SLO, define two alerts:

• Fast burn (outage): 5 min window consuming > 2% of the 30-day budget -> page.
• Slow burn (regression): 1 hour window consuming > 5% of the 30-day budget -> ticket, not page.

Example (crash-free sessions, SLO 99.5%, error budget 0.5%):

- alert: crash_free_sessions_fast_burn
  expr: (1 - crash_free_sessions_ratio{release="current"}) > 14.4 * 0.005
  for: 5m
  severity: page
- alert: crash_free_sessions_slow_burn
  expr: (1 - crash_free_sessions_ratio{release="current"}) > 3 * 0.005
  for: 1h
  severity: ticket

14.4x and 3x come from the Google SRE burn-rate table for 2%/1h and 5%/6h budget consumption.

3. Anti-Noise Rules

• Minimum impact threshold: do not page for issues affecting < 1% of sessions and < 100 sessions/hour.
• Deduplicate by release: if the same alert fired for the previous release and is still firing on the new one, annotate rather than re-page.
• OS / device floor: mute alerts that only fire on OS versions older than N-3 if your policy does not cover those.
• Auto-resolve: configure auto-resolution when the metric returns below threshold for 2x the evaluation window.
• Maintenance windows: suppress during planned staged rollouts, app review submission, or coordinated backend deploys.

4. Required Alert Fields

Every alert must carry:

• summary: one sentence, user-visible symptom.
• impact: "X% of users on release Y on platform Z".
• runbook_url: link to the markdown runbook (on-call-mobile).
• deep_links: one per relevant tool -- Crashlytics issue, Sentry dashboard, Datadog RUM session, trace.
• owner: team slack channel + PagerDuty service.

annotations:
  summary: "Crash-free sessions below SLO on [email protected]"
  impact: "1.2% of sessions affected over last 1h"
  runbook_url: "https://runbooks/crash-free-sessions"
  sentry_url: "https://sentry.io/my-org/my-app-android/?release=4.12.0"
  datadog_url: "https://app.datadoghq.com/rum/sessions?query=@release:4.12.0"

5. Deep Links Into Traces

Wire every alert into the trace that caused it:

• Include a panel link with the OTel query {service.name="my-app", error=true} filtered to the alert window.
• Link to one representative session replay (respecting privacy) so the on-call can see the user journey.
• For crash alerts, link to the Crashlytics/Sentry issue URL pre-scoped to the firing release.

6. Alert Ownership and Review

• Every alert has a single owning team. No "everyone" channels.
• Weekly alert review meeting: delete any alert that was acknowledged but did not lead to action in the last 30 days.
• Quarterly blameless audit: count pages per on-call rotation; if > 2 per shift on average, the rotation is overloaded.

7. Testing Alerts

• Wire a chaos job that forces a synthetic crash / slow start on a canary release, so you can validate the alert fires end-to-end at least once per quarter.
• Use prometheus/promtool test rules or the vendor's test-mode to unit-test alert expressions.

8. Examples of Good Alerts

- name: checkout_error_rate
  expr: sum(rate(checkout_payment_submit_count{outcome="error"}[5m]))
      / sum(rate(checkout_payment_submit_count[5m])) > 0.05
  for: 10m
  severity: page
  annotations:
    summary: "Checkout error rate > 5% over 10m"
    impact: "Users cannot pay"
    runbook_url: "https://runbooks/checkout"

- name: cold_start_p95_regression
  expr: histogram_quantile(0.95,
          sum(rate(app_start_cold_duration_ms_bucket{release="current"}[15m]))
          by (le, device_class))
        > on(device_class)
        1.5 * histogram_quantile(0.95,
          sum(rate(app_start_cold_duration_ms_bucket{release="previous"}[1h]))
          by (le, device_class))
  for: 30m
  severity: ticket

9. Examples of Alerts to Delete

• "Any new Sentry issue" -- create an issue auto-triage rule instead.
• "CPU > 80% on > 10 devices" -- not a user symptom.
• "Network latency > 500 ms" without endpoint grouping -- too broad.

Checklist

• Every paging alert corresponds to a user-visible symptom.
• Each SLO has both a fast-burn (page) and slow-burn (ticket) alert.
• Minimum impact threshold prevents paging on tiny cohorts.
• All alerts include summary, impact, runbook_url, and deep links.
• One team owns each alert; no shared channels.
• Alerts are reviewed weekly and tested quarterly.
• Dedup, auto-resolve, and maintenance-window handling are configured.