On call mobile

On-Call for Mobile

Instructions

Mobile on-call is different from backend on-call: you cannot hotfix in 5 minutes, store review takes hours to days, and users on the buggy build may be unreachable for weeks. Plan for slow fixes and fast mitigations.

1. Severity Levels

Sev	Definition	Response time	Escalation
SEV1	Widespread outage: > 5% crash-free session drop, or critical flow unusable for >10%	15 min	Immediate incident channel
SEV2	Partial outage: SLO fast-burn alert; critical flow degraded	30 min	On-call + team lead
SEV3	Single-feature regression; isolated issue with workaround	2 hours	On-call
SEV4	Tracked issue, no user impact yet	Next business	Backlog

2. The On-Call Role

• One primary and one secondary per rotation, per platform when possible.
• Shift length: 1 week, handover on Monday with a runbook review.
• Mobile on-call owns: crash alerts, ANR alerts, release health regressions, store status page.
• Mobile on-call does not own backend outages unless the mobile app cannot fall back to a safe state.

3. First 10 Minutes of a SEV1

1. Acknowledge the page.
2. Open the incident channel; post the firing alert's deep links.
3. Identify the affected release(s) and platform(s) from the alert context.
4. Decide: mitigate first, diagnose second.
   • Feature flag kill-switch off?
   • Remote config rollback?
   • Halt staged rollout (Play / App Store)?
   • Server-side workaround (return a safe payload that avoids the crash path)?
5. Publish a status message internally (and externally if public) with "investigating, users on release X.Y.Z may see ..."

4. Mitigation Tools (Have These Ready)

• Feature flags (LaunchDarkly, Firebase Remote Config, Statsig): every risky feature behind a flag with a documented kill-switch.
• Staged rollout controls: bookmark Play Console and App Store Connect pages for the app.
• Backend safe-mode: an endpoint that returns conservative defaults (empty feed, no experiments) for users on the affected release.
• Forced update screen: a bootstrap check that shows an "update required" UI if the version is in a published deny-list. Use sparingly.

5. Crash Spike Playbook

1. Identify the new issue in Crashlytics / Sentry; note the release and affected OS/device.
2. Check if the issue is on a new release (likely a new regression) or across releases (likely an external dependency, e.g. OS update).
3. If new release and rollout is not yet 100%: halt rollout immediately.
4. If 100%: push a hotfix. Meanwhile, feature-flag the offending code path off if possible.
5. Open a postmortem ticket with the Sentry/Crashlytics issue URL, the user count affected, and the mitigation taken.

6. ANR / Hang Spike Playbook

1. Look for the screen / operation in last_screen and last_operation custom keys.
2. Check for a main-thread I/O or heavy computation change in the recent diff.
3. If correlated with a server payload change, coordinate with the backend on-call to roll back or trim the payload.
4. If device-specific: add device class filter; if impact is small, ticket and fix; otherwise halt rollout.

7. Store Rejection / Removal

• Treat App Store or Play rejection as a SEV2.
• Obtain the rejection message; classify (policy, technical, metadata).
• If the rejection implicates functionality, not metadata, halt rollout and prepare a resubmission.
• Keep a list of reviewer contact emails and expedited-review criteria for SEV1 cases.

8. Rollback Strategies

Mobile "rollback" is not a single button:

• Client-only: disable a feature flag or remote config value.
• Backend fallback: server returns a safe payload structure.
• Staged rollout halt: stop promotion; no new users receive the bad build.
• Force update: gate the app on a minimum supported version; users below are forced to update. Use only when the risk of leaving users on the bad build exceeds the UX cost of forcing an update.
• Take-down: remove from the store listing. Users on the bad build keep the bad build until they update; this is rarely a useful mitigation alone.

9. Runbooks

Each SLO and high-cardinality error should have a runbook at runbooks/<topic>.md that includes:

• Summary of what the alert indicates.
• First-look dashboard and query links.
• Common causes (ranked).
• Mitigation steps (ranked by speed).
• Escalation contacts.
• Postmortem template link.

Runbooks are versioned in the app repo. Alerts must link to them.

10. Postmortems

• Within 3 business days of a SEV1/SEV2: written postmortem, blameless, with a timeline, contributing factors, impact estimate, and action items.
• Action items are tracked in the backlog with owners and due dates.
• Postmortems are shared across teams so backend on-call sees mobile failure modes and vice versa.

11. On-Call Hygiene

• Target: < 2 pages per shift on average. If > 2, tune alerts before rotating.
• Provide "learn while on-call" time: no feature work expected during a shift; focus on triage and runbook improvements.
• Backup support contacts (PagerDuty override) in case of emergency absence.

Checklist

• Severity matrix is documented with response times and escalation paths.
• Primary and secondary on-call exist per platform; shift length and handover are defined.
• Mitigation toolkit (feature flags, rollout controls, backend safe-mode) is documented and tested.
• Crash-spike, ANR-spike, and store-rejection playbooks exist and are linked from alerts.
• Rollback strategies are documented and chosen appropriately per incident.
• Runbooks are versioned and referenced by every alert.
• Postmortems are required for SEV1/SEV2 and action items tracked.
• Page budget per shift is monitored and alert noise is tuned.