agentsclimarketplace

Incident triage and pipeline recovery

Skill vaquarkhan/data-engineering-agent-skills/skills/incident-triage-and-pipeline-recovery

Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

Install
npx -y skills add vaquarkhan/data-engineering-agent-skills --skill incident-triage-and-pipeline-recovery

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 21 stars21 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Guides agents through production data incidents. Use when a pipeline fails, publishes bad data, misses an SLA, partially loads, corrupts state, or requires rollback, replay, or stakeholder communication.

SKILL.md

2.7 KB, 544 tokens by cl100k_base, as published. Nobody here has run it

Incident Triage And Pipeline Recovery

Overview

Use this skill when production behavior is already wrong and the team needs controlled recovery. It helps agents contain blast radius, diagnose quickly, restore trust, and avoid making the incident worse.

When to Use

  • failed production runs
  • bad or partial publishes
  • corrupted incremental state
  • freshness SLA breaches
  • emergency rollback or replay decisions

Do not jump to fixes before stabilizing the system and understanding impact.

Workflow

  1. Contain the incident. Decide whether to:

    • pause schedules
    • block downstream publish
    • isolate bad partitions
    • notify owners and consumers

  2. Pull live signals before mutation when possible. Load mcp-data-observability-integration to inspect lag, run state, or Spark stage metrics before reruns or replays.

  3. Classify impact. Identify:

    • affected datasets
    • time window
    • downstream consumers
    • whether data is late, missing, duplicated, or wrong

  4. Determine the safest recovery path. Options include:

    • rerun
    • replay
    • rollback
    • partial correction
    • full backfill

    When replay or backfill is chosen, load safe-backfill-and-replay-orchestration and draft templates/backfill-plan.yaml before execution.

  5. Validate recovery before reopening publish paths.

  6. Record the incident and add a guardrail. A fixed incident with no prevention work is unfinished.

  7. Turn high-value incidents into repeatable resilience tests. Load skills/data-resiliency-testing-and-failure-injection/SKILL.md or references/data-resiliency-testing-patterns.md when the same failure mode must be prevented from surprising the team again.

Common Rationalizations

RationalizationReality
"Let us rerun everything now."Blind reruns can duplicate data or destroy evidence.
"We can clean up downstream later."Downstream trust loss is often harder to recover than the pipeline itself.
"The root cause can wait."Without a guardrail, the same incident often returns quickly.

Red Flags

  • no attempt to contain blast radius
  • reruns happen before impact is understood
  • downstream consumers are not informed
  • recovery succeeds but no new guardrail is added

Verification

  • Impact scope and affected consumers are identified
  • The chosen recovery path is safer than the alternatives
  • Publish is reopened only after validation
  • Post-incident guardrails or checks are added

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.