Workflow planning

Workflow Planning

Overview

Turn an approved experiment design into a concrete execution plan with all checkpoint artifacts created.

Core principle: The plan creates the harness. After this skill completes, workflow-state.json, progress.log, and init.sh exist and are ready.

Process

Step 1: Read the Experiment Design Doc

Read the full design doc. Extract: objectives, stages, parameters, success criteria, pitfalls, software, walltimes.

Step 2: Define Concrete Stages

For each stage in the design, define:

Field	Content
id	stage-1, stage-2, stage-3, ... (always sequential integers)
name	Human-readable name
depends_on	List of stage IDs this depends on
inputs	Exact file paths (relative to workflow root) as array of strings
outputs	Expected output file paths as array of strings
parameters	Software-specific parameters from the design (object)
success_criteria	Specific, measurable criteria as a single string
backend	Backend profile name (optional — omit to use default_backend)

Stage ID rules:

• Always stage-N with sequential integers: stage-1, stage-2, stage-3, stage-4, stage-5
• Never use sub-labels like stage-2a, stage-2b — even for parallel branches
• If the design doc uses sub-labels (e.g., "Stage 2a, 2b, 2c"), renumber them to sequential integers
• Branching is expressed via depends_on, not via IDs

Output directory convention:

• Each stage writes outputs to a stage-N/ subdirectory
• Example: stage-2/isotherm.csv, stage-3/output.log
• Stage 1 may write to a named directory if outputs are shared inputs (e.g., structures/)

Step 2b: Ask About Autonomous Mode

After defining all stages, ask the user:

"This workflow has N stages. Enable autonomous session chaining? If yes, executing-workflows will launch a background runner that automatically chains Claude sessions until the workflow completes or blocks. Requires: tmux installed, tool permissions pre-approved in .claude/settings.json."

• If the user says yes → set session_config.autonomous: true in the workflow-state.json created in Step 3.
• If the user says no or skips → leave session_config.autonomous: false (the default).

Step 3: Create workflow-state.json

Create with all stages in pending status. Every stage MUST have all fields shown below — no optional fields, no extra fields.

{
  "workflow_id": "<topic>-YYYY-MM-DD",
  "version": 1,
  "created": "<ISO-timestamp>",
  "updated": "<ISO-timestamp>",
  "experiment_design": "<relative-path-to-design-doc>",
  "workflow_plan": "<relative-path-to-plan-doc>",
  "amendments": [],
  "session_config": {
    "autonomous": false,
    "session_budget": 6,
    "session_count": 0,
    "session_id": null,
    "session_cost": 0,
    "exit_reason": null,
    "stage_weights": {
      "sync": 1,
      "async": 1.5,
      "error_cycle": 2,
      "diagnostic": 2
    }
  },
  "default_backend": "local",
  "backend_profiles": {
    "local": {
      "type": "local",
      "config": {
        "batch_type": "Shell",
        "context_type": "LocalContext"
      }
    },
    "hpc-cluster": {
      "type": "remote",
      "batch_type": "Slurm",
      "config_path": "/home/user/.dpdisp/hpc_config.json",
      "resource_defaults": {
        "number_node": 1,
        "cpu_per_node": 16,
        "gpu_per_node": 0,
        "queue_name": "",
        "group_size": 1
      }
    }
  },
  "stages": [
    {
      "id": "stage-1",
      "name": "Human-readable name",
      "status": "pending",
      "depends_on": [],
      "backend": null,
      "inputs": ["path/to/input.file"],
      "outputs": ["stage-1/output.file"],
      "parameters": {
        "software": "SoftwareName",
        "key_param": "value"
      },
      "success_criteria": "Specific measurable criterion in a single string",
      "started_at": null,
      "completed_at": null,
      "retry_count": 0,
      "last_error": null,
      "running_process": null
    }
  ]
}

Schema rules:

• stages is an array, not an object
• inputs and outputs are arrays of strings (file paths), not objects
• success_criteria is a single string, not an array
• parameters is an object with key-value pairs
• Every stage includes ALL operational fields (started_at, completed_at, retry_count, last_error, running_process) initialized to null/0
• Top-level MUST include version: 1 and amendments: []
• Top-level MUST include default_backend and backend_profiles (at minimum a "local" profile)
• Stage backend field is optional — null means use default_backend
• backend_profiles with type: "remote" must include batch_type (the scheduler type from the machine config, e.g., "Slurm", "PBS", "LSF", "SGE"), config_path (path to external machine config — validated for existence, never read), and resource_defaults
• Top-level MUST include session_config with session_budget (default 6), session_count (0), session_id (null), session_cost (0), exit_reason (null), and stage_weights (sync: 1, async: 1.5, error_cycle: 2, diagnostic: 2). session_count is incremented by session-resume at each session start — single source of truth for session numbering. User may adjust session_budget and stage_weights after planning.
• session_config.autonomous is a boolean (default false). Set to true during planning when the user opts in to autonomous session chaining.
• Do NOT add extra fields (description, known_pitfalls, safeguards, expected_walltime, execution_order) — those belong in the design doc or plan doc, not the state file

Step 4: Create progress.log

Single line via Bash tool. No more.

echo "[$(date -Iseconds)] Workflow created: <workflow_id>. N stages. Purpose: <purpose>." > progress.log

Note: > (create) not >> (append) — this is the initial creation.

Step 5: Create init.sh

Build environment checks for every software tool in the workflow. This script runs at every session resume — if it's wrong, the workflow breaks.

Critical rules:

• Use set -e — fail immediately on any error
• Use direct conda env paths, not conda activate (unreliable in non-interactive shells)
• Every check MUST exit 1 on failure — never use WARNING for required software
• Check every software binary, every Python library, every tool

Template:

#!/bin/bash
set -e

# ── Infrastructure (always required) ──
command -v uvx >/dev/null 2>&1 || { echo "FAIL: uvx not found"; exit 1; }
command -v tmux >/dev/null 2>&1 || { echo "FAIL: tmux not found"; exit 1; }
uvx --from dpdispatcher dpdisp --help > /dev/null 2>&1 || { echo "FAIL: dpdispatcher not accessible via uvx"; exit 1; }

# ── Project-specific (derive ALL from design doc) ──
# Use direct conda/venv paths — `conda activate` is unreliable in non-interactive shells.
# Every check MUST `exit 1` on failure — never WARNING for required software.
#
# Examples (adapt to your software stack):
# CONDA_ENV="/path/to/conda/env"
# PYTHON="$CONDA_ENV/bin/python3"
# $PYTHON --version || { echo "FAIL: python3 not found at $PYTHON"; exit 1; }
# $PYTHON -c "import numpy; import torch" || { echo "FAIL: required Python libraries not available"; exit 1; }
# command -v lmp >/dev/null 2>&1 || { echo "FAIL: LAMMPS not found"; exit 1; }
# command -v simpleFoam >/dev/null 2>&1 || { echo "FAIL: OpenFOAM not found"; exit 1; }

echo "Environment ready."

For each software in the design doc, add a check:

• Computation binaries (e.g., LAMMPS, OpenFOAM, VASP, GROMACS, custom solvers): check binary path
• Python/R libraries (e.g., numpy, torch, MDAnalysis, pandas, scikit-learn): check import
• External tools (e.g., AutoPoly, samtools, PLUMED, ffmpeg): check binary if used in scripts

Step 6: Write Plan Doc

Save to docs/superscientist/plans/YYYY-MM-DD-<topic>.md:

# [Topic] Workflow Plan

**Experiment Design:** [relative path to design doc]
**Workflow ID:** [workflow_id]
**Stages:** N total

## Stage Execution Order

1. Stage 1: [name] — [brief description]
2. Stage 2: [name] (depends on: stage-1) — [brief description]
...

## Per-Stage Details

### Stage N: [Name]
- **Dependencies:** [list]
- **Inputs:** [file paths]
- **Commands:** [exact commands to run — not vague descriptions]
- **Outputs:** [expected files]
- **Success criteria:** [specific]
- **Estimated walltime:** [from design]
- **Backend:** [profile name, e.g., "local" or "hpc-cluster"]
- **Dispatch mode:** [sync (local < 2 min) or async (tmux)]

Commands must be exact. Not "Run the simulation at 298K" but the actual command: lmp -in input.lammps or python train.py --config config.yaml. If the exact command depends on stage preparation, describe the command template and what the subagent fills in.

Backend and dispatch: Each stage specifies which backend profile to use. The superscientist:compute-backend skill handles dispatch — sync for local short jobs (< 2 min), async via tmux for everything else (remote backends always, local long jobs).

Step 7: Commit All Artifacts

git add workflow-state.json progress.log init.sh docs/superscientist/plans/
git commit -m "feat: create workflow plan for <topic>"

Step 8: Transition

Announce: "Workflow plan created. Invoke superscientist:executing-workflows to begin execution, or review the plan first."

Red Flags

Thought	Reality
"I'll create the state files later"	Create them now. They ARE the plan.
"init.sh can be generic"	Add software-specific checks from the design. Every tool.
"Dependencies are obvious"	Write them explicitly in depends_on.
"I'll use stage-2a/2b/2c for parallel branches"	Always sequential integers. Branching is in depends_on.
"conda activate works fine"	Not in non-interactive shells. Use direct paths.
"A WARNING is enough for missing software"	Hard fail (exit 1). Session-resume depends on this.
"Commands can be vague in the plan"	Exact commands. The executing agent needs them.