Pipeline replay

Pipeline Replay

The two worst sentences in automation: "it ran twice" and "it died on step four, how do I restart it." Both have the same root cause. Nobody decided, per step, what is safe to repeat and what is not. So the retry sends the email again, or you can't resume without redoing the charge.

This skill makes that decision explicit, step by step, before it bites. The pattern it builds is the one I run in production: a persisted state file is the source of truth, every step saves to it, and a re-run skips finished work by looking at the data, not a flag. You describe the pipeline, it produces a replay plan.

Work through every step. Build the plan, then stop.

Inputs (ask for whatever is missing)

• The pipeline (required): the steps in order, what each one does. If a build-spec exists, take the happy path and the state notes from it.
• What each step touches: which steps call an external system (send, charge, post, write to someone else's database) vs only read or write your own state.
• Optional: where state lives (a JSON file, a DB row, a queue), and the real incident if this is a post-mortem ("it sent two emails").

The method

1. List the steps in order. One row per step. If a step does two things to the outside world, split it — a row should have one external effect at most.

2. Classify each step's side effect. The core of the whole skill:

   • read — pure lookup. Always safe to repeat.
   • internal-write — writes only your own state. Safe to repeat if it's an upsert on a stable key.
   • external-write — sends/posts/writes to a system you don't own (email, Slack, a CRM, a webhook). Repeating it is visible to someone else. Needs a dedup key.
   • irreversible — charges, deletes, sends something you can't unsend. Repeating it is real damage. Needs a dedup key and a pre-check.

3. Make state the source of truth, and save after every step. Name the one place that records what's done (a state file, a DB row with a status field, the record itself). The rule is to persist after each step, not once at the end — a pipeline with a single checkpoint at the end can only restart from zero.

4. Skip finished work by filtering the data, not by flags. A re-run should look at the state and skip what's already there: leads.filter(l => !l.score), "process posts not in processedPosts", "rows where email is null". Presence of the output is the done-marker. This is what makes re-running a no-op for completed work without a separate bookkeeping layer that can drift.

5. Assign an idempotency key per writing step. What makes "the same work" the same: prospect_id + campaign_day, invoice_id, order_id + step. You check it before acting and store it after. No key means no safe retry — design one in here.

6. Find the safe resume point. For each failure step, name the step the pipeline can re-enter from without repeating an external-write or irreversible action. That is the answer to "it died on step four." If there isn't one, that's the gap to design out now.

7. Build the dedup table. One row per external-write / irreversible action: the action, its idempotency key, where the key is recorded, and the check that runs before acting ("if a send exists for this key, skip"). This table is what turns a re-run into a no-op instead of a second send.

8. Set the unattended guardrails. Run it single-flight (concurrency limit 1) so two copies never overlap on the same state. If it loops toward a target, give the loop a hard iteration cap and a break-when-no-progress, so a stuck source can't spin forever. On error, aggregate failures into the run's result and return them rather than throwing the whole run away. Then the retry/backoff and the kill switch.

Output

Produce the replay plan as two markdown tables — a step ledger (step, side-effect class, idempotency key, saves-state?, safe resume point) and a dedup table (action, key, recorded where, pre-check) — followed by the guardrail rules (single-flight, loop cap, error-aggregation, kill switch). Call out, in one line each, any step that has no safe resume point or no dedup key: those are the bugs to fix before this runs unattended.

Then stop. The plan is the deliverable. Wiring the keys and checks into the code is the next step, and now you know exactly which steps need them.