Incident response

Incident Response

Handle production incidents from triage through postmortem. Phase A creates incident tickets, correlates recent deployments, and recommends mitigation actions. Phase B produces structured postmortem reports with root cause analysis and improvement tracking.

Capability Dependencies

• Phase A (Incident Triage):
  • One tracker provider skill such as jira or github-issue
  • One repo provider skill such as bitbucket, github, or gitlab
  • One ci provider skill such as jenkins or github-workflow
• Phase B (Postmortem):
  • One tracker provider skill (configured)
  • A docs provider skill such as confluence or obsidian
  • One repo provider skill (optional, for linked commit/PR evidence)

Inputs

Phase A — Incident Triage

Variable	Required	Description	Example
TRACKER_PROJECT	Yes	Selected tracker project/container key for the incident ticket (Jira project key, or GitHub Issue owner/repo)	PROJ
INCIDENT_SUMMARY	Yes	One-line incident description	Payment service returning 500 errors
SEVERITY	Yes	Incident severity: P1 (critical), P2 (high), P3 (medium), P4 (low)	P1
REPO_TARGETS	Conditional	Comma-separated <provider>:<namespace>/<repo> targets for change correlation; only when deployment investigation is needed	bitbucket:sample-org/sample-service
CI_JOB	Conditional	CI provider job/workflow path for the deployment being investigated; only when deployment correlation is needed	folder-a/deploy-prod
INCIDENT_TYPE	No	Tracker issue type for incident tickets (default: Incident)	Bug
SUBAGENT_MODE	No	Follows the shared definition in parallel-dispatch.md (default: auto)	auto
MAX_CONCURRENT_COMMITS	No	Follows the shared MAX_CONCURRENT_<UNIT> convention in parallel-dispatch.md; unit = Commits (default: 4)	4

Phase B — Postmortem

Variable	Required	Description	Example
INCIDENT_KEY	Yes	Tracker issue key for the incident to document	PROJ-123
DOCS_SPACE	Conditional	Selected docs space/container (Confluence space key, or Obsidian top-level folder); only when publishing	PROJ
DOCS_PARENT_REF	Conditional	Parent page or folder identifier for postmortem reports; only when publishing	1531709516
CROSS_REVIEW_COUNT	No	Number of independent cross-review subagents for postmortem quality (default: 5)	5
RE_REVIEW_SEVERITY	No	Severity threshold for adversarial re-review of postmortem findings: high, medium (default), all	medium

Follow the shared missing-input stop rule in ../using-orbit/references/safety-rules.md.

Scenario-specific input rule:

• Phase A can start with TRACKER_PROJECT, INCIDENT_SUMMARY, and SEVERITY
• ask for REPO_TARGETS and CI_JOB only when the user wants deployment change correlation
• Phase B can start with INCIDENT_KEY alone
• ask for DOCS_SPACE and DOCS_PARENT_REF only when the user wants docs publication

Missing-input collection rules

• Ask for missing inputs in one plain-text message, requesting only the smallest still-missing set for the current phase.
• Use normalized shared field names from ../using-orbit/references/common-input-contract.md whenever the input belongs to the shared capability contract (TRACKER_PROJECT, REPO_TARGETS, DOCS_SPACE, DOCS_PARENT_REF).
• INCIDENT_KEY, INCIDENT_SUMMARY, SEVERITY, and INCIDENT_TYPE are scenario-specific business inputs. CI_JOB is a provider-native route input scoped to the selected ci provider.

Critical Prompt-Shape Override

When the user reports a production incident and provides TRACKER_PROJECT and INCIDENT_SUMMARY, treat it as sufficient to start Phase A. Make the first live stop boundary the tracker/repo/ci preflight — if config is missing, return only the exact missing values and stop.

If the user says "复盘" or "postmortem" and provides an INCIDENT_KEY, start directly in Phase B.

If the user says "生产事件" or "incident" without specifying, detect from intent:

• "P1 事件", "故障响应", "出问题了" → Phase A (triage)
• "复盘", "postmortem", "事后总结" → Phase B (postmortem)
• Ambiguous → ask which phase

Before You Start

Before resolving the first stop boundary, read only the Missing-Input Stop Rule section of ../using-orbit/references/safety-rules.md.

Preflight Configuration Check

This skill has two phases with different capability requirements:

• Phase A (triage): follow the preflight protocol in ../using-orbit/references/safety-rules.md for the selected tracker, repo, ci providers.
• Phase B (postmortem): follow the same protocol for the selected tracker, docs providers in addition to the already-resolved context. repo is optional in Phase B (used only when linked commit/PR evidence is needed).

After boundary resolved

Read ../using-orbit/references/safety-rules.md, ../using-orbit/references/cli-patterns.md, and ../using-orbit/references/output-conventions.md. Read ../using-orbit/references/parallel-dispatch.md before dispatching per-commit investigation workers. Read ../using-orbit/references/cross-review.md only when entering postmortem cross-review / re-review stages. Use those shared references for common execution policy. This skill file remains responsible for the scenario-specific two-phase incident/postmortem flow and change correlation logic.

Phase A: Incident Triage (故障响应)

Step 1: Create Incident Ticket (requires user confirmation)

Follow ../using-orbit/references/safety-rules.md write confirmation protocol.

Display the incident ticket details before creating.

Jira example:

jira issue create --project {{TRACKER_PROJECT}} --type {{INCIDENT_TYPE}} --summary "[{{SEVERITY}}] {{INCIDENT_SUMMARY}}" --description "{{INCIDENT_SUMMARY}}" --fields-json '{"priority": "{{PRIORITY_MAP}}", "labels": ["incident", "{{SEVERITY}}-severity"]}'

GitHub Issue example (for GitHub Issue, TRACKER_PROJECT carries the owner/repo form; split into --namespace/--repo):

github-issue issue create --namespace {{OWNER}} --repo {{REPO}} --title "[{{SEVERITY}}] {{INCIDENT_SUMMARY}}" --description "{{INCIDENT_SUMMARY}}" --labels "incident,{{SEVERITY}}-severity"

The description field uses INCIDENT_SUMMARY as initial content. If the user provides additional details at this point, append them to the description.

Priority mapping (Jira):

• P1 → Highest
• P2 → High
• P3 → Medium
• P4 → Low

If the Incident issue type does not exist in the project, fall back to Bug with the incident label. Note this fallback in the output.

After creation, capture INCIDENT_KEY for subsequent steps.

Step 2: Correlate Recent Deployments (conditional)

Only execute when CI_JOB is provided or when REPO_TARGETS is available.

CI deployment correlation:

Check the most recent build:

jenkins build get --job {{CI_JOB}} --number lastCompletedBuild
github-workflow build get --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --number <LATEST_RUN>

Check the previous successful build for comparison:

jenkins build get --job {{CI_JOB}} --number lastSuccessfulBuild

Extract: build number, result, timestamp, branch, version parameters. If the last completed build is FAILURE or the time between the last successful build and the incident report is short, flag this deployment as a suspect.

Repo change correlation:

When REPO_TARGETS is available, search for recently merged PRs. For each target, parse provider/namespace/repo.

Bitbucket example:

bitbucket pr list --project {{REPO_NAMESPACE}} --state MERGED --after {{INCIDENT_WINDOW_START}}

GitHub example (no MERGED list state — list closed and filter by merged_at):

github pr list --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --state closed

GitLab example:

gitlab pr list --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --state merged

Set INCIDENT_WINDOW_START to 24 hours before the incident report time (or a user-specified window). This captures recent changes that may have caused the incident.

Each recently merged PR is a potential change suspect. For each, read details:

Bitbucket example:

bitbucket pr get --project {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --id {{PR_NUMBER}}

GitHub example:

github pr get --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --number {{PR_NUMBER}}

Parallel Per-Commit Investigation

Each recently merged PR is independent. Follow the shared dispatch rules in ../using-orbit/references/parallel-dispatch.md.

• Work unit: One PR → read diff, assess risk.
• Batch size: MAX_CONCURRENT_COMMITS (default 4).

For each suspect PR:

Bitbucket example:

bitbucket pr diff --project {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --id {{PR_NUMBER}}

GitHub example:

github pr diff --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --number {{PR_NUMBER}}

GitLab example:

gitlab pr diff --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --number {{PR_NUMBER}}

Assess whether the change could have caused the reported symptoms. Rank suspects by likelihood.

Step 3: Build Incident Timeline

Assemble a chronological timeline from collected evidence:

1. Deployment event: build #, timestamp, version deployed
2. Suspect changes: PR titles, merged timestamps, authors
3. Incident reported: INCIDENT_KEY creation time
4. Current status: service health (from SMOKE_TEST_URLS if provided, otherwise note as unknown)

Output the timeline in the incident ticket as a comment.

Jira example:

jira issue comment add --key {{INCIDENT_KEY}} --text "{{TIMELINE_TEXT}}"

Follow ../using-orbit/references/safety-rules.md write confirmation protocol.

Step 4: Recommend Mitigation Actions

Based on the incident severity and correlated changes, recommend one or more mitigation actions:

Scenario	Recommendation
Last deployment likely caused the incident	Rollback: redeploy the last known good version via the ci provider
Specific PR identified as suspect	Revert PR: create a revert PR in the repo provider (requires confirmation)
Root cause unclear	Investigate: read ci console logs and build parameters for more evidence
Service is down and no rollback path	Escalate: add escalation-needed label and notify the on-call team

Rollback action (requires user confirmation):

Jenkins example:

jenkins job build-with-params --job {{CI_JOB}} --params-json '{"BRANCH":"<PREVIOUS_BRANCH>","VERSION":"<PREVIOUS_VERSION>"}'

GitHub Workflow example:

github-workflow job build --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --workflow {{CI_JOB}} --ref <PREVIOUS_BRANCH>

Follow ../using-orbit/references/safety-rules.md write confirmation protocol before triggering any rollback build.

Phase A Output

## Incident {{INCIDENT_KEY}} — Triage Summary

### Ticket
- Key: {{INCIDENT_KEY}}
- Severity: {{SEVERITY}}
- Summary: {{INCIDENT_SUMMARY}}

### Deployment Correlation
| Time | Event | Details |
|------|-------|---------|
| 2026-06-12 09:30 | Deploy | Build #42 (v1.2.0) — SUCCESS |
| 2026-06-12 10:15 | Incident | {{INCIDENT_KEY}} reported |

### Suspect Changes (last 24h)
| PR | Repo | Title | Risk | Author |
|----|------|-------|------|--------|
| #42 | sample-service | Auth refresh flow | HIGH | user1 |
| #44 | sample-service | Cache config update | MEDIUM | user2 |

### Recommended Mitigation
1. Rollback to v1.1.0 (Build #41) — reverses both suspect changes
2. Alternative: revert PR #42 only — targeted fix

### Action Required
- Confirm rollback to proceed, or investigate further

Phase B: Postmortem (复盘归档)

Prerequisite: The incident should be resolved or in a stable state. This phase produces the postmortem report after the immediate incident is handled.

Step 1: Collect Incident Evidence

Read the full incident ticket.

Jira example:

jira issue get --key {{INCIDENT_KEY}}
jira issue comment list --key {{INCIDENT_KEY}}

Find linked commits and PRs. Tracker↔repo linkage is provider-specific; follow ../using-orbit/references/provider-linking-strategy.md for the selected pair. Bitbucket provides direct tracker↔commit linkage:

Bitbucket example:

bitbucket jira issue-commits --issue-key {{INCIDENT_KEY}}

For other repo providers without a native tracker-link command, derive linked commits from the incident ticket's remote links and repo PR search, then correlate by commit SHA.

For each linked commit:

bitbucket commit pull-requests --project {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --id <COMMIT_ID>
gitlab commit pull-requests --namespace {{REPO_NAMESPACE}} --repo {{REPO_NAME}} --sha <COMMIT_SHA>

If the incident has subtasks, read them:

jira issue subtasks --key {{INCIDENT_KEY}}

Step 2: Generate Structured Postmortem

From the collected evidence, generate a structured postmortem report following this template:

**Postmortem: {{INCIDENT_KEY}} — {{INCIDENT_SUMMARY}}**

- **Severity**: P1/P2/P3/P4
- **Duration**: <start time> → <end time> (<total duration>)
- **Impact**: <user-facing impact description>
- **Root Cause**: <code-level or infrastructure-level root cause>
- **Trigger**: <what specifically triggered the incident>
- **Resolution**: <how the incident was resolved>
- **Detection**: <how the incident was detected — monitoring alert, user report, etc.>

**Timeline**
| Time | Event |
|------|-------|
| <timestamp> | <event description> |

**Suspect Changes**
| PR | Repo | Author | Assessment |
|----|------|--------|------------|
| #42 | sample-service | user1 | Confirmed root cause |

**Action Items**
1. <improvement item> — Owner: <assignee> — Tracker: <ticket key if created>
2. <improvement item> — Owner: <assignee> — Tracker: <ticket key if created>

**Lessons Learned**
- <lesson 1>
- <lesson 2>

Step 2b: Cross-Review Postmortem Quality (交叉评审)

After the initial postmortem is generated, dispatch cross-review subagents to independently validate postmortem quality.

Follow the cross-review dispatch, merge, and retention rules in ../using-orbit/references/cross-review.md.

• Work unit: One review lens → independently evaluate the full postmortem and underlying evidence.
• Batch size: CROSS_REVIEW_COUNT (default 5).
• Inline fallback: Process lenses sequentially when dispatch unavailable.

Cross-review lenses:

Lens	Focus	Key Question
root-cause-evidence	Root cause grounding	Is the root cause supported by code/infrastructure evidence, or is it speculative?
timeline-completeness	Timeline gaps	Are there unexplained gaps between the deployment event and incident detection?
impact-accuracy	Impact assessment	Is the user-facing impact accurately described with concrete evidence?
action-item-actionability	Improvement tracking	Are action items concrete, assigned, and trackable — or vague and unowned?
recurrence-prevention	Prevention effectiveness	Would the proposed action items actually prevent recurrence?

Merge: Apply cross-review merge rules. Findings agreed by ≥ 2 cross-reviewers are CONFIRMED. Single-reviewer findings are UNIQUE. Direct contradictions are CONFLICT and surfaced to the user.

Step 2c: Re-Review Postmortem Findings (复审)

For each cross-review finding selected by RE_REVIEW_SEVERITY, plus all UNIQUE findings, dispatch a re-review verifier subagent that attempts to refute the finding.

Follow the re-review dispatch, verdict, and retention rules in ../using-orbit/references/cross-review.md.

Each verifier returns one of: CONFIRMED, REFUTED, or ADJUSTED.

After re-review: Remove REFUTED findings. Apply ADJUSTED findings to the final postmortem.

Step 3: Publish Postmortem to Docs (requires user confirmation)

Only enter this step when the user wants docs writeback. If DOCS_SPACE or DOCS_PARENT_REF is missing, ask for them here.

Follow the preflight protocol in ../using-orbit/references/safety-rules.md for the selected docs provider before continuing.

Follow ../using-orbit/references/safety-rules.md docs deduplication protocol. Before creating, search for existing postmortems.

Confluence example:

confluence search content --cql "title ~ '{{INCIDENT_KEY}}' AND space = '{{DOCS_SPACE}}'" --type page --limit 5

Obsidian example:

obsidian content search --query "{{INCIDENT_KEY}}"

If a matching page/note is found, follow the shared docs deduplication protocol.

Follow ../using-orbit/references/safety-rules.md write confirmation protocol.

Display the full postmortem content first, then after confirmation:

Confluence create example:

confluence page create --space {{DOCS_SPACE}} --title "Postmortem: {{INCIDENT_KEY}} {{INCIDENT_SUMMARY}}" --parent-id {{DOCS_PARENT_REF}} --body "{{POSTMORTEM_HTML}}"

Confluence update example:

confluence page update --id <EXISTING_PAGE_ID> --title "Postmortem: {{INCIDENT_KEY}} {{INCIDENT_SUMMARY}}" --body "{{POSTMORTEM_HTML}}" --version <NEXT_VERSION>

Obsidian create example:

obsidian content create --space {{DOCS_SPACE}} --title "Postmortem: {{INCIDENT_KEY}} {{INCIDENT_SUMMARY}}" --body "{{POSTMORTEM_MARKDOWN}}" --parent-ref {{DOCS_PARENT_REF}}

Step 4: Create Improvement Tracker Tickets (requires user confirmation)

For each action item from the final postmortem, offer to create a tracking ticket.

Follow ../using-orbit/references/safety-rules.md write confirmation protocol.

Display the planned tickets before creating. After confirmation:

Jira example:

jira issue create --project {{TRACKER_PROJECT}} --type Task --summary "<action item summary>" --description "<action item details from postmortem>" --fields-json '{"labels": ["postmortem-action", "{{INCIDENT_KEY}}"]}'

GitHub Issue example:

github-issue issue create --namespace {{OWNER}} --repo {{REPO}} --title "<action item summary>" --description "<action item details from postmortem>" --labels "postmortem-action,{{INCIDENT_KEY}}"

Link each action ticket to the incident.

Jira example:

jira issue remote-link add --key {{INCIDENT_KEY}} --url "<TRACKER_TICKET_URL>" --title "Improvement: <action item summary>"

Local Persistence

Local state is stored under ~/.config/incident-response/:

~/.config/incident-response/
  runs/
    <incident-key>.json

Run state shape:

{
  "incidentKey": "PROJ-123",
  "severity": "P1",
  "phase": "A",
  "phaseA": {
    "ticketCreated": true,
    "deployCorrelation": {},
    "suspectChanges": [],
    "mitigationRecommended": null
  },
  "phaseB": {
    "postmortemGenerated": false,
    "docsPageId": null,
    "actionItems": []
  }
}

Internal local-state persistence under ~/.config/incident-response/ is allowed without extra confirmation.

Allowed Automatic Writes

• Local state persistence under ~/.config/incident-response/
• HTTP health check requests (read-only GET, no side effects)
• Tracker issue comment add for timeline (after confirmation)

Forbidden Automatic Writes

• CI build trigger without confirmation (including rollback builds)
• repo pr merge / pr decline / revert PR without confirmation
• Tracker issue create without confirmation
• Tracker issue transition apply without confirmation
• docs page create / update without confirmation and dedup check
• Any SonarQube or Fortify changes