agentsclimarketplace

Debug a production incident

Skill protosphinx/sphinxstack/skills/debug-a-production-incident

A stack of skills for your AI agent, made for people starting from zero. 103 skills and 149 project briefs — load one into Codex, Claude Code, Copilot, Gemini, or Cursor and do the thing.

Install
npx -y skills add protosphinx/sphinxstack --skill debug-a-production-incident

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Restore a production service while preserving the evidence needed to understand the failure. Establish impact, contain harm, test the recovery, and record follow-up work. Use when real users are affected or a production signal requires coordinated investigation.

SKILL.md

3.6 KB, 645 tokens by cl100k_base, as published. Nobody here has run it

debug-a-production-incident

Run an incident as two connected jobs: reduce harm now, then learn enough to prevent a repeat. Prefer safe restoration over a clever diagnosis while users are still affected. Keep one factual timeline so responders do not have to reconstruct the event from chat.

When to use

Use this skill when users are failing, data may be wrong, an important service-level signal is breached, or responders must coordinate a production investigation. Do not declare an incident for a local development bug with no shared or production effect.

Never expose customer data, credentials, or private logs in a public channel. Do not make several uncontrolled changes at once; each change needs an owner, timestamp, and observed result.

Preconditions

  • Open an incident record with a lead, communications owner, severity, and start time.
  • Identify the safest source of production signals and the authority required for changes.
  • Preserve relevant logs, traces, deploy identifiers, feature-flag state, and recent change history.
  • Agree on a communication interval and the audience for internal and external updates.

Procedure

  1. State the user-visible symptom, affected population, first known time, and current severity.
  2. Start an incident timeline. Record observations and actions separately from hypotheses.
  3. Check broad dependencies and recent changes before diving into one component.
  4. Choose the safest containment: disable a feature, shed optional work, fail over, limit traffic, revert a configuration, or rollback a release. Predict the signal that should improve.
  5. Make one controlled change. Record who made it, when, why, and what happened next.
  6. Confirm recovery from the user's path as well as internal dashboards. A green process is not enough when checkout, login, or data delivery still fails.
  7. Continue monitoring through a defined stability window. Keep the incident open while errors are recurring, queues are draining, or data repair is incomplete.
  8. Once stable, narrow the causal chain with timestamps and evidence. Distinguish trigger, contributing conditions, failed defenses, and user impact.
  9. Hold a post-incident review without blame. Create only follow-ups that have an owner, priority, verification method, and due condition.
  10. Publish an accurate final update and store the incident record where future responders can find it.

Failure plan

Prepare for a containment action that makes the incident worse. Define the rollback command, the signal that triggers it, and the person authorized to execute it. If evidence conflicts, return to the last confirmed observation. If data integrity or security may be involved, stop destructive repair and involve the responsible owner before proceeding.

Done

  • An incident timeline separates observations, hypotheses, decisions, and production changes
  • The impact window names affected users, functions, and duration without unsupported precision
  • Recovery proof includes the user journey, system signals, and a completed stability window
  • The post-incident review identifies trigger, contributing conditions, and failed defenses
  • Every corrective action has a follow-up owner and an explicit verification method

Then use design-observability to repair missing signals and create-a-deployment-pipeline if release controls failed.

What ships with it: 1 file

237 B alongside SKILL.md

agents/

Gives 0 of the 12 instructions most debug triage skills give in 645 tokens

Counted across 839 of the 1,149 authors here whose files we hold, read 2026-08-07

  • Investigate root cause before proposing any fixin 102 of 839, across 67 files
  • Read error messages completelyin 89 of 839, across 49 files
  • Create a failing test case before fixingin 84 of 839, across 46 files
  • Reproduce the issue consistentlyin 82 of 839, across 41 files
  • Change one variable at a timein 82 of 839, across 42 files
  • Check recent changesin 74 of 839, across 36 files
  • Write the regression test before fixingin 74 of 839, across 40 files
  • Fix the root cause not the symptomin 60 of 839, across 45 files
  • Implement a single fix at a timein 59 of 839, across 20 files
  • Trace data flow backward to the sourcein 50 of 839, across 20 files
  • Remove all debug instrumentationin 49 of 839, across 13 files
  • Form a single hypothesisin 48 of 839, across 18 files

Said here and by no other author read

  • prefer safe restoration over clever diagnosis
  • open an incident record with lead severity and start time
  • preserve logs traces and recent change history
  • record observations separately from hypotheses
  • confirm recovery from the user path
  • keep the incident open while errors recur

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 326,984. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.