Executing analysis plans

Executing Analysis Plans

Overview

A plan that's been approved is a commitment, and execution is where it either gets honored or quietly abandoned. This skill takes over once question-framing (and, for confirmatory work, pre-analysis-plan) have produced an approved plan, and carries it out: build, estimate, stress-test, assemble — including dispatching independent work to parallel subagents.

Core principle: Execute the approved plan faithfully, validating as you go and parallelizing what's independent. Autonomy here is for carrying out the agreed plan fast and thoroughly, not for changing it — any departure is a checkpoint, not a step.

Prerequisite: there is an approved plan

Don't start here from a cold "analyze this." If there's no approved brief/PAP yet, go back to question-framing (and pre-analysis-plan for confirmatory work) first — executing a plan nobody approved is just the behind-the-back problem wearing a schedule.

A new request on an already-locked plan still triggers this skill — re-fire, don't coast. A re-run, a finer reporting cut, "now do the other radii / the facility-year version" is exactly this skill's job, and verification is still worth offering before any result is written to a file (result-verification runs only if the user asks). "The design was locked last week, I'll just run it" is how a new cut ships unverified — the lock covers the design, not this run. If the new cut changes the unit or estimand, it's a question-framing/analysis-checkpoints change first, not a re-run.

The sequential spine vs. the parallel fan-out

The single biggest execution mistake is running everything in one slow serial loop — or, worse, parallelizing things that actually depend on each other. Split the plan into its dependent spine and its independent leaves.

Sequential spine (must run in order — each depends on the last):

1. Build / clean / join the analysis dataset. Unless it's a trivial load of one already-clean file, this is a phase, not a line: delegate it to data-preparation, which validates each cleaning step (data-contracts) and routes consequential decisions to analysis-checkpoints. Nothing downstream is trustworthy until that phase's reconciliation passes.
2. Construct the treatment, outcome, and key covariates → validate ranges, missingness, leakage.
3. Estimate the primary specification (the one pre-committed in the PAP) → this is the number.

Parallel fan-out (independent — but chosen, not exhaustive): Once the validated dataset and primary spec exist, the supporting analyses are independent and can run concurrently — but "can run in parallel" is not "should all run." Pick the shortlist that earns its place (next section) and fan out only those, one subagent per task. Candidates to choose from:

• a robustness specification that probes the main threat (not every control permutation);
• the placebo / falsification test that would catch the confound you actually worry about;
• a genuinely alternative design, when one exists;
• a pre-specified subsample / heterogeneity cut (not a fishing sweep);
• the secondary outcome the mechanism predicts;
• the one sensitivity analysis that matters (Oster δ, e-value, bandwidth);
• for structural work: the Monte-Carlo recovery reps, the multiple starting values on the non-convex objective, and the counterfactual scenarios (one per mechanism) — same fan-out logic (structural-estimation).

But choosing the execution mode is itself a checkpoint — present it, don't assume it — unless a topology has already decided it.

An approved phase IS the execution-mode consent when it carries a topology. If an approved docs/analysis/phases/<id>.yaml already has a non-empty topology.nodes (schema owned by analysis-state-management, the normal case under causal-conductor on OpenCode), the user already made the inline-vs-fan-out call: one leaf node per independent piece of work is the fan-out plan, pre-approved. Map the shortlist onto those nodes and dispatch — don't re-ask, that's friction dressed up as diligence. The ask-first rule below applies only when no topology exists yet.

When no topology has already decided it, ask the user how they want the shortlist run:

• Inline — each chosen spec in turn, here in this session, every step fully observable as it goes.
• Subagent dispatch — one parallel subagent per spec, run concurrently.

Recommend subagent dispatch for genuinely independent specs (faster, and it isolates each spec's context from yours), but it is the user's call — inline keeps everything in view, which some prefer for a small or delicate run. Make this a real, up-front question, not a silent default.

Once the mode is settled: for subagent dispatch, hand each chosen task to the robustness-runner agent Causal Powers ships — it executes one pre-specified spec, asserts the data contracts, and returns a structured result, stopping if it hits a design decision (superpowers' dispatching-parallel-agents / subagent-driven-development cover the mechanics). A robustness-runner dispatch is one topology.nodes leaf-node dispatch, just named differently by layer — one node per call, never a multi-node prompt, the collapse the topology exists to prevent. For inline, run the chosen specs one at a time, applying the same data-contract assertions to each.

Robustness is an argument, not an inventory

The parallel machinery makes running checks cheap, and cheap is exactly the trap: it turns the robustness suite into a free buffet. Don't. More robustness checks do not mean more credibility — a wall of specifications usually buries the one check that matters, and a senior reader treats a 30-column robustness table as a tell of weak identification, not a show of strength. Before running anything beyond the primary spec:

1. Name the main threat — the single most credible way this estimate could be confounding rather than effect.
2. Choose the ~3 checks that would actually break the result if it's fragile — the ones that probe that threat. A good robustness check has a real chance of failing; a cosmetic "add one more control" that cannot fail proves nothing.
3. Propose the shortlist to the user — each with a one-line rationale ("drop the cities with the 2016 recording jump — tests whether the recording change, not the policy, drives it") — and get approval before running.
4. Run only the approved set. Run more only if the user asks. (If the robustness suite was already locked in pre-analysis-plan, run that set as-is — the checkpoint is for additions or deviations, not a redundant re-ask of the pre-approved plan.)

Default to roughly three. This is judgment, not a quota — occasionally a design genuinely needs a fourth, and you should say so — but the instinct is parsimony, because the job is to convince, not to exhaust. Choosing which checks to run is itself a consequential decision, so it goes through the user (analysis-checkpoints), never a silent fan-out of everything imaginable.

What every dispatched subagent must carry

A parallel subagent is a place for silent errors and silent redesigns to hide, so constrain it:

• The exact, pre-specified task — the precise spec/test from the approved plan, not "explore X." It executes a recipe; it does not choose the recipe.
• The data contracts to assert — the same data-contracts invariants, so a fanned-out spec can't quietly run on a corrupted subset.
• A structured result to return — coefficient, SE, N, the diagnostics, and a pass/fail on its contracts — so you can assemble them without re-reading ten transcripts.
• The checkpoint rule — if it hits a decision that would change the design, sample, spec, or estimand, it reports back and stops rather than resolving it; that decision routes to the user via analysis-checkpoints.

Between every step: validate, then checkpoint

Execution is not "run to the end and show the user." After each spine step and as fan-out results land:

• Validate the result against its contract (data-contracts); reconcile totals; if a number looks wrong, switch to wrong-number-debugging.
• Checkpoint any consequential decision that surfaced (analysis-checkpoints) — execution is exactly when "the data surprised us, let's change the design" arises, and that's the user's call, not a step you take to keep moving.

Keep durable state live, and compact at phase boundaries

A long analysis with many mid-step fixes drags context until auto-compaction fires at a random, lossy moment, losing gotchas and decisions you can't afford to lose. Don't wait for that: actively maintain the plan/brief/model card as you go — mark steps done, record the gotcha you just hit — a living document, not something you wrote once at the start.

Make the trigger mechanical, not a vibe: run the update-and-offer-compact routine after each completed spine step and after the fan-out is assembled (the checkpoints the skill already defines), not whenever a phase "feels" done. At each:

1. Update docs/analysis/ so it stands on its own — invoke analysis-state-management and update whichever records changed (see its own "How To Update State" for which file). State lives in the repo, not the chat, and it must carry the decisions locked and why, the key insight so far, and the concrete next step as a resume-from-clean-slate instruction.
2. Offer to compact: "this is a clean point to /compact — docs/analysis/index.yaml points to the decisions, insight, and next step, so we resume on a clean slate without losing anything." You can't compact yourself (the user runs /compact), so suggest it — at real phase boundaries only, never mid-step, and easy to wave off.
3. On resume, follow analysis-state-management's Resume Rule: index.yaml first, then only the records it names, continuing from next_action.

The test: if the conversation were compacted right now, could a clean session pick up from docs/analysis/index.yaml plus the named records alone? If not, the state isn't finished — write the missing state before you suggest the compact.

Synthesis

When the fan-out completes, assemble — don't just dump:

• Build the robustness table: primary estimate beside every alternative, so stability (or fragility) is visible at a glance.
• Reconcile across specs: if the headline swings under a reasonable alternative, that's a finding to surface, not a result to bury.
• Note which subagents' contracts failed — a robustness spec that violated an invariant is not a clean "it's robust."
• Before any of this is reported, offer verification (result-verification) — running it is the user's call.

Red flags — STOP

• Starting execution with no approved plan to execute.
• Fanning out an exhaustive menu of robustness checks instead of proposing the ~3 that probe the main threat and getting approval first. More checks ≠ more credibility.
• Parallelizing steps that actually depend on each other (e.g. estimating before the dataset is validated).
• A subagent that resolved a design/sample/spec decision on its own instead of reporting it back.
• Improvising new specifications mid-execution that weren't in the plan, without surfacing them.
• Presenting fanned-out results without reconciling them or checking each one's contracts.

Common rationalizations

Excuse	Reality
"More robustness checks make it more convincing."	Less — a wall of specs reads as theater and hides the one check that matters. Pick the few that could actually break the result.
"Running them is cheap now, so why not run them all?"	Cheap-to-run is the trap; the cost is the reader's trust and the buried signal. Propose ~3 and get approval.
"The subagents can figure out the spec."	An under-specified subagent invents its own analysis — deciding behind the user's back, in parallel. Hand each one the exact recipe.
"A robustness check failed its data contract, but the coefficient looks fine."	A spec run on corrupted data isn't robustness, it's noise. The contract failing is the result.
"The data suggested a better spec, so I added it."	That's specification search. Surface it as a checkpoint; run it labeled exploratory if approved.
"I'll show all the results at the end."	A wrong intermediate poisons everything after it unseen. Validate each step as it lands.

The Process

1. Build / clean / join the dataset → delegate to data-preparation (unless it's a trivial load of one already-clean file); it returns the clean, validated dataset.
2. Validate every later spine step + fanned-out spec before trusting a number → invoke data-contracts.
3. Fork by design as you estimate — reduced-form → invoke causal-identification for the robustness suite; structural → invoke structural-estimation for recovery reps, starts, and counterfactuals.
4. Keep the code minimal and surgical on every write → invoke analysis-craft.
5. Any decision that changes design/sample/spec/estimand — or any number the user has already seen — STOP and invoke analysis-checkpoints. A surprising result is a checkpoint, not a step; a number that looks wrong is wrong-number-debugging, not a silent patch.
6. Spine + fan-out complete → offer verification (result-verification) in one line — the user's call. result-verification is user-invoked; offer it before a result ships, never run it unasked.

The bottom line

Executing well  →  approved plan worked top to bottom, spine validated in order, independent specs fanned out to subagents, every result reconciled, deviations stopped for the user
Otherwise        →  a serial half-run of an unapproved plan, with the robustness suite quietly truncated