Text orchestrator

Text Orchestrator (Automated Pipeline Runner)

Automatically run through text pipeline stages by invoking subskills in sequence.

This skill is the automated orchestrator for the text pipeline. It coordinates:

• workspace initialization via text-intake
• sequential stage execution via stage-specific skills
• state tracking and advancement
• blocker detection and graceful stopping

It does not own:

• any stage's artifact generation directly
• semantic evaluation decisions
• human approval (Stage 10 always requires explicit approval)

Environment

Text data lives in the texts repository. Set LYCEUM_TEXTS_DIR to point to this directory (defaults to output/texts for local development).

Quick Status

Pipeline workspaces: !find ${LYCEUM_TEXTS_DIR:-output/texts} -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l Workspaces in progress: !find ${LYCEUM_TEXTS_DIR:-output/texts} -path '*/state.json' -exec grep -l '"status": "pending"\|"status": "in_progress"' {} \; 2>/dev/null | wc -l

Commands

• /text-orchestrator run [work] [--mode add|apply] [--profile <profile>] — Run the full pipeline for a text
• /text-orchestrator replay [work] --profile <profile> — Replay specific stages on an existing workspace
• /text-orchestrator resume [work] — Resume an in-progress pipeline from where it stopped
• /text-orchestrator step [work] — Execute only the next pending stage, then stop
• /text-orchestrator status [work] — Show pipeline progress and next stage
• /text-orchestrator abort [work] — Mark the pipeline as blocked and stop

Target: $ARGUMENTS

---

Stage-to-Skill Mapping

Stage ID	Stage Name	Skill to Invoke
0-intake	Intake	text-intake
1-source-discovery	Source Discovery	source-hunt
2-acquisition-extraction	Acquisition	source-extract
3-cleaning-normalization	Cleaning	text-cleaning
4-structural-segmentation	Segmentation	segmentation
5-witness-normalization-ranking	Witness Ranking	translation-witness
6-versification-alignment	Versification	versify try-witness → (if fail) translation-synthesis → versify
7-transliteration	Transliteration	transliteration
8-interlinear-morphology	Interlinear	treebank → alignment → gloss-review
9-reader-reliability	Reader QA	reader-reliability
10-human-review	Ship Gate	new-text-ship (requires human approval)

---

Orchestration Protocol

Before Running Any Stage

Step 0. Create workspace backup (MANDATORY):

SLUG="your-text-slug"
STAGE="N-stage-name"
cp -r "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}" "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-${STAGE}-$(date +%Y%m%d-%H%M%S)"

1. Load workspace state:

   nix-shell -p jq --run "jq '.' ${LYCEUM_TEXTS_DIR:-output/texts}/SLUG/state.json"
   

2. Check current stage status and blockers:

   nix-shell -p jq --run "jq '{current: .current_stage, invalidated: .invalidated_stages, stages: [.stages[] | select(.requested) | {stage: .stage, status: .status}]}' ${LYCEUM_TEXTS_DIR:-output/texts}/SLUG/state.json"
   

3. Check manifest for blockers:

   nix-shell -p jq --run "jq '.blockers' ${LYCEUM_TEXTS_DIR:-output/texts}/SLUG/manifest.json"
   

Stage Execution Loop

For each stage, the orchestrator must:

1. Backup the workspace (see "Defense Mechanisms" section)
2. Announce the stage being executed
3. Read the stage skill to understand its requirements
4. Invoke the skill's run command with safety constraints and timeout
5. Verify workspace integrity (CRITICAL — see "Defense Mechanisms"):
   • Workspace directory still exists
   • manifest.json exists and is valid JSON
   • state.json exists and is valid JSON
6. If integrity check fails: restore from backup, mark failed, STOP
7. Verify the stage completed successfully by checking:
   • Stage status updated to done or blocked
   • Expected outputs exist
   • No new blockers were added
8. Update workspace state if the skill didn't do so
9. Advance to the next stage or stop if blocked
10. Cleanup old backups (keep last 3)

Stopping Conditions

The orchestrator MUST stop when:

• A stage sets status to blocked
• A stage fails verification
• Manifest blockers are non-empty
• Stage 10 is reached (requires human approval)
• An unrecoverable error occurs

---

Workflows

/text-orchestrator run

Full automated pipeline execution from intake through ship gate.

Steps

1. Initialize workspace (if not exists):

   Load skill: text-intake
   Execute: /text-intake init [work] --mode [mode] --profile [profile]
   Verify: manifest.json and state.json exist
   

2. Enter stage loop:

   WHILE next_stage exists AND next_stage != "10-human-review":
     stage = determine_next_stage()
     skill = map_stage_to_skill(stage)
     
     ANNOUNCE: "=== Executing Stage: {stage} via {skill} ==="
     
     Load skill file for {skill}
     Execute: /{skill} run [work]
     
     IF stage has sub-skills (e.g., Stage 6, Stage 8):
       Execute each sub-skill in order
     
     Verify stage completion
     
     IF blocked OR failed:
       STOP with report
   

3. Reach ship gate:

   ANNOUNCE: "Pipeline reached Stage 10 (Human Review)"
   ANNOUNCE: "Automatic execution complete. Human approval required."
   Execute: /new-text-ship preview [work]
   STOP (do not auto-promote)
   

Multi-Skill Stages

Stage 6 (Versification) — conditional fast path:

1. /versify try-witness [work] — Attempt to versify a versification-candidate witness directly
2. If try-witness succeeds: Stage 6 is done. Skip Stage 6a generation entirely. Run verify_stage_6b.sh (the versified witness edition satisfies the same structural contract as a generated translation).
3. If try-witness fails (no candidate, or all candidates fail versification): a. /translation-synthesis run [work] — Generate translation (Stage 6a) b. Run verify_stage_6a.sh c. /versify run [work] — Validate and import (Stage 6b) d. Run verify_stage_6b.sh

Stage 8 (Interlinear) — with sub-stage tracking:

Stage 8 consists of 4 canonical sub-stages:

• treebank — Generate/import treebank (optional)
• llm-interlinear — Generate contextual glosses via LLM
• ground-truth-benchmark — Benchmark against external corpora
• gloss-review — Audit and promote

Progress announcement:

=== Stage 8: Interlinear & Morphology ===
  → Sub-stage 1/4: Treebank import (optional)
  → Sub-stage 2/4: LLM interlinear generation (est. ~5min/100 words in agent mode)
  → Sub-stage 3/4: Ground-truth benchmark
  → Sub-stage 4/4: Gloss review & promotion

Sub-stage 1: Treebank (optional)

SLUG="your-text-slug"
WORKSPACE="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"

# Check if already done
STATUS=$(nix-shell -p jq --run "jq -r '.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.treebank.status // \"not_started\"' ${WORKSPACE}/state.json")

if [[ "$STATUS" == "done" ]]; then
  echo "Sub-stage treebank already done, skipping"
else
  echo "→ Sub-stage 1/4: Treebank import"
  START_TIME=$(date +%s)
  
  # Mark in progress
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.treebank) = {status: \"in_progress\", started_at: \"$(date -Iseconds)\"}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
  
  # Run treebank skill (may skip if no treebank available)
  /treebank run "${SLUG}"
  
  END_TIME=$(date +%s)
  ELAPSED=$((END_TIME - START_TIME))
  
  # Mark done
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.treebank) = {status: \"done\", completed_at: \"$(date -Iseconds)\", elapsed_seconds: ${ELAPSED}, notes: [\"Treebank import completed\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
fi

Sub-stage 2: LLM Interlinear (DO NOT use text_pipeline_alignment.go)

# Check if already done
STATUS=$(nix-shell -p jq --run "jq -r '.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"llm-interlinear\".status // \"not_started\"' ${WORKSPACE}/state.json")

if [[ "$STATUS" == "done" ]]; then
  echo "Sub-stage llm-interlinear already done, skipping"
else
  echo "→ Sub-stage 2/4: LLM interlinear generation"
  START_TIME=$(date +%s)
  
  # Mark in progress
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"llm-interlinear\") = {status: \"in_progress\", started_at: \"$(date -Iseconds)\"}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
  
  # Use API mode if key available (5-10x faster), fall back to agent mode
  if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
    INTERLINEAR_MODE="api"
  else
    INTERLINEAR_MODE="agent"
  fi
  python3 scripts/generate_workspace_interlinear.py --workspace ${WORKSPACE} --mode ${INTERLINEAR_MODE} -v
  
  END_TIME=$(date +%s)
  ELAPSED=$((END_TIME - START_TIME))
  
  # Mark done
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"llm-interlinear\") = {status: \"done\", completed_at: \"$(date -Iseconds)\", elapsed_seconds: ${ELAPSED}, notes: [\"LLM interlinear generation completed\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
fi

Sub-stage 3: Ground-truth Benchmark

# Check if already done
STATUS=$(nix-shell -p jq --run "jq -r '.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"ground-truth-benchmark\".status // \"not_started\"' ${WORKSPACE}/state.json")

if [[ "$STATUS" == "done" ]]; then
  echo "Sub-stage ground-truth-benchmark already done, skipping"
else
  echo "→ Sub-stage 3/4: Ground-truth benchmark"
  START_TIME=$(date +%s)
  
  # Mark in progress
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"ground-truth-benchmark\") = {status: \"in_progress\", started_at: \"$(date -Iseconds)\"}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
  
  # Benchmark against external corpora
  go run ./scripts/text_pipeline_ground_truth.go benchmark -root ${LYCEUM_TEXTS_DIR:-output/texts} -work ${SLUG}
  
  END_TIME=$(date +%s)
  ELAPSED=$((END_TIME - START_TIME))
  
  # Mark done
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"ground-truth-benchmark\") = {status: \"done\", completed_at: \"$(date -Iseconds)\", elapsed_seconds: ${ELAPSED}, notes: [\"Ground-truth benchmark completed\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
fi

Sub-stage 4: Gloss Review

# Check if already done
STATUS=$(nix-shell -p jq --run "jq -r '.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"gloss-review\".status // \"not_started\"' ${WORKSPACE}/state.json")

if [[ "$STATUS" == "done" ]]; then
  echo "Sub-stage gloss-review already done, skipping"
else
  echo "→ Sub-stage 4/4: Gloss review & promotion"
  START_TIME=$(date +%s)
  
  # Mark in progress
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"gloss-review\") = {status: \"in_progress\", started_at: \"$(date -Iseconds)\"}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
  
  # Run benchmark and promote
  go run ./scripts/text_pipeline_gloss_review.go benchmark -root ${LYCEUM_TEXTS_DIR:-output/texts} -work ${SLUG}
  go run ./scripts/text_pipeline_gloss_review.go promote -root ${LYCEUM_TEXTS_DIR:-output/texts} -work ${SLUG}
  
  END_TIME=$(date +%s)
  ELAPSED=$((END_TIME - START_TIME))
  
  # Mark done
  nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"gloss-review\") = {status: \"done\", completed_at: \"$(date -Iseconds)\", elapsed_seconds: ${ELAPSED}, notes: [\"Gloss review and promotion completed\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json
fi

On resume: The orchestrator checks each sub-stage's status before executing. If a sub-stage is marked done, it skips it entirely.

On failure: Mark the failed sub-stage as blocked or failed with notes:

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages.\"llm-interlinear\") = {status: \"blocked\", completed_at: \"$(date -Iseconds)\", notes: [\"LLM API rate limit exceeded\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

Clearing sub-stages (e.g., when replaying Stage 8):

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"8-interlinear-morphology\") | .sub_stages) = {}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

CRITICAL for Stage 8: The LLM pipeline (generate_workspace_interlinear.py) produces contextual glosses like "wrath", "him", "walking". The Go-based pipeline produces dictionary-style glosses like "to be", "he, she, it; self". ALWAYS use the Python LLM pipeline. Also, gloss-review expects the workspace ground-truth benchmark artifact to already exist, so run text_pipeline_ground_truth.go benchmark before text_pipeline_gloss_review.go benchmark/promote.

Performance expectations:

• Agent mode (--mode agent): ~5 minutes per 100 words (subprocess cold-start dominates)
• API mode (--mode api): ~30 seconds per 100 words (5-10x faster, requires ANTHROPIC_API_KEY)
• Section splitting: Sections >200 words are automatically split by llm_interlinear.py to maintain reasonable batch sizes
• Recommendation: For texts with >500 total words, set ANTHROPIC_API_KEY and use --mode api to avoid excessive pipeline overhead
• Typical wall time (agent mode): A 1600-word text (e.g., Meditations Book 1) takes ~90 minutes
• Sub-stage timing: Track elapsed time for each sub-stage to identify bottlenecks

---

/text-orchestrator replay

Replay specific stages on an existing workspace by invalidating and re-executing them according to a replay profile.

Purpose

Use when you need to:

• Upgrade source files for a shipped text
• Redo cleaning with improved rules
• Regenerate segmentation after fixing bugs
• Refresh witnesses or versification
• Rebuild transliteration with new standards
• Regenerate interlinear with improved LLM prompts
• Re-run reader QA after fixes
• Completely rehab a text

Key difference from run: Replay expects the workspace to already exist and uses replay profiles to determine which stages to invalidate and re-execute.

Key difference from resume: Resume picks up where a pipeline stopped. Replay explicitly invalidates stages via a profile before executing, even if those stages were previously marked done.

Available Replay Profiles

Profile	Stages Replayed	Downstream Invalidated	Use When
source-upgrade	0,1,2,3,4	Everything from stage 4 onward	Better Greek source found
cleaning	0,2,3	Everything from stage 3 onward	Cleaning rules improved
segmentation	0,4	5,6,7,8,9,10	Segmentation logic fixed
witness-upgrade	0,1,5	6,8,9,10	New PD witnesses available
versification	0,4,5,6,9,10	-	Versification logic changed
transliteration	0,7,9,10	-	Transliteration standard updated
interlinear	4,5,6,7,8	5,6,7,8,9,10	Regenerate stages 4-8 from existing source data
treebank-enrichment	0,8,9,10	-	Treebank data upgraded
reliability	0,9,10	-	Reader QA criteria changed
full-rehab	All stages	All stages	Complete text rehabilitation

Note on invalidation cascades: The table shows the requested replay stages. Invalidation rules in textpipeline.go may mark additional downstream stages as invalidated, making them actionable too. For example, segmentation profile requests stage 4, but stage 4 invalidates 5,6,7,8,9,10 — so all of those become actionable and will execute during the replay.

Steps

1. Verify workspace exists:

   SLUG="your-text-slug"
   WORKSPACE="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"
   
   if [[ ! -d "${WORKSPACE}" ]]; then
     echo "ERROR: Workspace not found at ${WORKSPACE}"
     echo "Use '/text-orchestrator run' to create a new workspace"
     exit 1
   fi
   
   if [[ ! -f "${WORKSPACE}/manifest.json" ]] || [[ ! -f "${WORKSPACE}/state.json" ]]; then
     echo "ERROR: Workspace missing critical files (manifest.json or state.json)"
     exit 1
   fi
   

2. Plan the replay via text-replay:

   Load skill: text-replay
   Execute: /text-replay plan [work] --profile [profile]
   
   Verify:
   - Profile matches one of the available profiles
   - Requested stages are identified
   - Downstream invalidations are computed
   - Current state is understood
   

3. Apply the replay plan:

   Execute: /text-replay run [work] --profile [profile]
   
   This updates:
   - state.json (marks replay stages as 'pending', downstream as 'invalidated')
   - replay/stage-history.json (appends replay event)
   - Sets current_stage to the first actionable stage
   

4. Enter the stage execution loop (same as run, but starting from first actionable stage):

   WHILE next_actionable_stage exists AND next_actionable_stage != "10-human-review":
     stage = determine_next_actionable_stage()  # Any stage with status 'pending' or 'invalidated'
     skill = map_stage_to_skill(stage)
     
     ANNOUNCE: "=== Replaying Stage: {stage} via {skill} ==="
     
     # Follow exact same protocol as /text-orchestrator run:
     # 1. Create backup (MANDATORY)
     # 2. Load skill file
     # 3. Spawn subagent with safety constraints and timeout
     # 4. Verify workspace integrity
     # 5. Run verification script
     # 6. Update state
     # 7. Advance or stop
     
     IF blocked OR failed:
       STOP with report
   

5. Reach ship gate or stop:

   IF current_stage == "10-human-review":
     ANNOUNCE: "Replay reached Stage 10 (Human Review)"
     ANNOUNCE: "Automatic execution complete. Human approval required."
     Execute: /new-text-ship preview [work]
     STOP (do not auto-promote)
   ELSE IF blocked:
     ANNOUNCE: "Replay stopped at stage {stage}: {blocker_reason}"
     STOP
   ELSE:
     ANNOUNCE: "Replay complete. All actionable stages executed."
     STOP
   

Actionable Stage Logic

A stage is actionable if its status is:

• pending (explicitly requested by the replay profile)
• invalidated (downstream of a replayed stage)

The orchestrator executes actionable stages in canonical order (0 → 10), regardless of which profile triggered them.

This logic is already implemented in internal/textpipeline/textpipeline.go via nextActionableStage().

Safety Protocols

Replay follows ALL the same defense mechanisms as run:

• Pre-stage backup (MANDATORY)
• Post-stage integrity verification
• Recovery from backup on integrity failure
• Subagent safety constraints
• Stage timeouts
• Verification script execution
• State updates and history logging
• Backup cleanup (keep last 3)

See "Defense Mechanisms" section for full details.

Example Replay Transcript

User: /text-orchestrator replay "plato-republic-book-1" --profile interlinear

Orchestrator: === Verifying Workspace ===
Workspace found: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1
Manifest: plato-republic-book-1 (add mode)
Current state: Stage 8 done, Stage 9 done, Stage 10 pending

=== Planning Replay ===
Loading skill: text-replay
Executing: /text-replay plan "plato-republic-book-1" --profile interlinear

Replay Profile: interlinear
Requested stages: 4, 5, 6, 7, 8
Invalidated stages: 5, 6, 7, 8, 9, 10 (downstream from Stage 4)
Actionable stages: 4 (pending), 5 (invalidated), 6 (invalidated), 7 (invalidated), 8 (invalidated)

=== Applying Replay Plan ===
Executing: /text-replay run "plato-republic-book-1" --profile interlinear

Updated state.json:
  - Stages 4-8 marked as pending/invalidated
  - current_stage set to 4-structural-segmentation

Appended to replay/stage-history.json:
  - Replay event: profile=interlinear, at=2026-03-22T14:30:00Z

=== Entering Stage Execution Loop ===

=== Replaying Stage: 4-structural-segmentation via segmentation ===
Creating backup: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1.backup-4-structural-segmentation-20260322-143000
Loading skill: segmentation
Executing: /segmentation run "plato-republic-book-1"

Stage 4 complete.
Integrity check: PASSED
Verification script: verify_stage_4.sh PASSED

=== Replaying Stage: 5-witness-normalization-ranking via translation-witness ===
Creating backup: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1.backup-5-witness-normalization-ranking-20260322-143100
Loading skill: translation-witness
Executing: /translation-witness run "plato-republic-book-1"

Stage 5 complete.
Integrity check: PASSED
Verification script: verify_stage_5.sh PASSED

=== Replaying Stage: 6-versification-alignment via versify ===
Creating backup: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1.backup-6-versification-alignment-20260322-143200

Stage 6 complete.
Integrity check: PASSED
Verification script: verify_stage_6b.sh PASSED

=== Replaying Stage: 7-transliteration via transliteration ===
Creating backup: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1.backup-7-transliteration-20260322-143300

Stage 7 complete.
Integrity check: PASSED
Verification script: verify_stage_7.sh PASSED

=== Replaying Stage: 8-interlinear-morphology via treebank → alignment → gloss-review ===
Creating backup: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-republic-book-1.backup-8-interlinear-morphology-20260322-143400

[Stage 8 multi-skill execution: treebank, LLM interlinear, ground-truth benchmark, gloss-review]

Stage 8 complete.
Integrity check: PASSED
Verification script: verify_stage_8.sh PASSED

=== Replay Complete ===
All actionable stages (4-8) executed successfully.
Stages 9-10 remain invalidated — run full pipeline or reliability profile to re-verify.

Common Replay Scenarios

Scenario 1: Source Upgrade

Problem: Found a better Greek source for an existing text
Solution: /text-orchestrator replay [work] --profile source-upgrade
Effect: Stages 0-4 re-run, stages 5-10 invalidated and re-executed

Scenario 2: Interlinear Improvement

Problem: Improved LLM prompt for glossing
Solution: /text-orchestrator replay [work] --profile interlinear
Effect: Stages 4-8 regenerated from existing source data, stages 9-10 invalidated but not executed

Scenario 3: Transliteration Standard Change

Problem: Updated transliteration rules repo-wide
Solution: /text-orchestrator replay [work] --profile transliteration
Effect: Stage 7 re-runs, stages 9-10 invalidated (but stages 5-6 and 8 are NOT invalidated)

Scenario 4: Complete Rehab

Problem: Text needs to be rebuilt from scratch (but keep original sources)
Solution: /text-orchestrator replay [work] --profile full-rehab
Effect: All stages re-execute from 0 through 10

Pre-Replay Checks

Before starting a replay, verify:

1. Workspace integrity:

   bash scripts/verify_workspace_integrity.sh "${SLUG}"
   

2. No concurrent operations:

   nix-shell -p jq --run "jq -r '.current_stage, .stages[] | select(.status == \"in_progress\")' ${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}/state.json"
   

   If any stage shows in_progress, either resume or abort it first.

3. Profile appropriateness:

   • Don't use full-rehab unless you really mean it
   • Use the most specific profile that matches your intent
   • Check invalidation cascade: text-replay plan shows what will be invalidated

Error Handling

Same as run, plus:

Workspace Not Found:

ERROR: Workspace not found at ${LYCEUM_TEXTS_DIR:-output/texts}/[slug]
ACTION: Use '/text-orchestrator run' to create a new workspace, not replay.

Invalid Profile:

ERROR: Unknown replay profile '[profile]'
ACTION: Use one of: source-upgrade, cleaning, segmentation, witness-upgrade, 
        versification, transliteration, interlinear, treebank-enrichment, 
        reliability, full-rehab

Stage Currently In Progress:

ERROR: Stage [stage] is currently in_progress
ACTION: Resume or abort the current operation before replaying.

---

/text-orchestrator resume

Resume a pipeline that was stopped.

Steps

1. Load workspace and determine current state
2. Find the next actionable stage
3. Continue the stage loop from that point
4. Follow same stopping conditions as run

Pre-Resume Checks

• Verify workspace exists
• Check if any stages are blocked (require manual intervention)
• Check if pipeline is already complete

---

/text-orchestrator step

Execute exactly one stage, then stop.

Use Cases

• Testing a single stage
• Manual intervention between stages
• Debugging pipeline issues

Behavior

Same as run but exits after one stage completes (success or failure).

---

/text-orchestrator status

Report pipeline progress without executing anything.

Output Format

Pipeline Status: [work]
==================
Workspace: ${LYCEUM_TEXTS_DIR:-output/texts}/[slug]
Mode: [add|apply]
Profile: [profile]

Stage Progress:
  [✓] 0-intake (done)
  [✓] 1-source-discovery (done)
  [→] 2-acquisition-extraction (in_progress)
  [ ] 3-cleaning-normalization (pending)
  ...

Next Stage: 2-acquisition-extraction
Next Skill: source-extract
Blockers: [none | list]

---

/text-orchestrator abort

Mark pipeline as blocked and stop.

Steps

1. Update state.json to mark current stage as blocked
2. Add note explaining why
3. Append history event

---

State Management

After Each Stage

Update state.json to reflect:

• Stage status: done, blocked, or failed
• last_run_at timestamp
• Outputs produced
• Notes from execution

Individual pipeline scripts (e.g., text_pipeline_segmentation.go) handle state updates internally. For manual state updates, use jq or direct file editing.

Appending History

Every stage execution should append to replay/stage-history.json:

{
  "at": "2026-03-17T19:00:00Z",
  "stage": "1-source-discovery",
  "action": "stage_completed",
  "notes": "Found 3 Greek candidates, 2 English witnesses"
}

Sub-Stage Tracking

Complex stages (e.g., Stage 6, Stage 8) consist of multiple sub-stages that can be tracked independently. The state model supports sub-stage tracking via the sub_stages map in each stage's state.

Sub-stage state structure:

{
  "stages": [
    {
      "stage": "8-interlinear-morphology",
      "status": "in_progress",
      "sub_stages": {
        "treebank": {
          "status": "done",
          "started_at": "2026-03-23T10:00:00Z",
          "completed_at": "2026-03-23T10:05:00Z",
          "elapsed_seconds": 300,
          "notes": ["Treebank import completed"]
        },
        "llm-interlinear": {
          "status": "in_progress",
          "started_at": "2026-03-23T10:05:00Z"
        }
      }
    }
  ]
}

Canonical sub-stage names:

• Stage 6: try-witness, translation-synthesis, versify
• Stage 8: treebank, llm-interlinear, ground-truth-benchmark, gloss-review

Updating sub-stage status (since scripts/text_pipeline_state.go doesn't exist, use jq):

Mark sub-stage in progress:

SLUG="your-text-slug"
STAGE="8-interlinear-morphology"
SUB="treebank"
WORKSPACE="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"${STAGE}\") | .sub_stages.\"${SUB}\") = {status: \"in_progress\", started_at: \"$(date -Iseconds)\"}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

Mark sub-stage done:

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"${STAGE}\") | .sub_stages.\"${SUB}\") = {status: \"done\", completed_at: \"$(date -Iseconds)\", elapsed_seconds: ${ELAPSED}, notes: [\"Sub-stage completed\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

Mark sub-stage blocked/failed:

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"${STAGE}\") | .sub_stages.\"${SUB}\") = {status: \"blocked\", completed_at: \"$(date -Iseconds)\", notes: [\"Reason for blocking\"]}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

Clear all sub-stages (when replaying a stage):

nix-shell -p jq --run "jq '(.stages[] | select(.stage == \"${STAGE}\") | .sub_stages) = {}' ${WORKSPACE}/state.json" > tmp.json && mv tmp.json ${WORKSPACE}/state.json

Resume logic: Before executing a sub-stage, check its status:

STATUS=$(nix-shell -p jq --run "jq -r '.stages[] | select(.stage == \"${STAGE}\") | .sub_stages.\"${SUB}\".status // \"not_started\"' ${WORKSPACE}/state.json")

if [[ "$STATUS" == "done" ]]; then
  echo "Sub-stage ${SUB} already done, skipping"
  # Skip to next sub-stage
else
  # Execute sub-stage
fi

Benefits:

• Fine-grained progress tracking within long-running stages
• Ability to resume from within a stage after interruption
• Detailed timing data for performance analysis
• Clear visibility into which sub-step is blocking

---

Error Handling

Stage Skill Not Found

ERROR: No skill mapping for stage [stage]
ACTION: Stop and report. This indicates a gap in the skill inventory.

Stage Execution Failed

ERROR: Stage [stage] failed with: [error]
ACTION: Mark stage as blocked, record error, stop orchestration.

Verification Failed

WARNING: Stage [stage] claims done but verification failed
ACTION: Mark as blocked with verification failure note, stop.

Human Intervention Required

INFO: Stage [stage] requires human decision
ACTION: Stop and report what decision is needed.

---

Defense Mechanisms

Pre-Stage Backup

Before invoking ANY stage skill, the orchestrator MUST create a backup:

# Create timestamped backup before stage execution
SLUG="your-text-slug"
STAGE="N-stage-name"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-${STAGE}-${TIMESTAMP}"

# Full workspace backup
cp -r "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}" "${BACKUP_DIR}"

# Verify backup was created
if [[ ! -d "${BACKUP_DIR}" ]]; then
  echo "FATAL: Failed to create backup at ${BACKUP_DIR}"
  exit 1
fi

echo "Backup created: ${BACKUP_DIR}"

Post-Stage Integrity Verification

After EVERY stage completes (success, failure, or timeout), verify workspace integrity:

SLUG="your-text-slug"
WORKSPACE="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"

# Critical checks
INTEGRITY_OK=true

# 1. Workspace directory exists
if [[ ! -d "${WORKSPACE}" ]]; then
  echo "CRITICAL: Workspace directory deleted!"
  INTEGRITY_OK=false
fi

# 2. manifest.json exists and is valid JSON
if [[ ! -f "${WORKSPACE}/manifest.json" ]]; then
  echo "CRITICAL: manifest.json missing!"
  INTEGRITY_OK=false
elif ! nix-shell -p jq --run "jq '.' '${WORKSPACE}/manifest.json'" > /dev/null 2>&1; then
  echo "CRITICAL: manifest.json is not valid JSON!"
  INTEGRITY_OK=false
fi

# 3. state.json exists and is valid JSON
if [[ ! -f "${WORKSPACE}/state.json" ]]; then
  echo "CRITICAL: state.json missing!"
  INTEGRITY_OK=false
elif ! nix-shell -p jq --run "jq '.' '${WORKSPACE}/state.json'" > /dev/null 2>&1; then
  echo "CRITICAL: state.json is not valid JSON!"
  INTEGRITY_OK=false
fi

if [[ "${INTEGRITY_OK}" != "true" ]]; then
  echo "INTEGRITY CHECK FAILED - INITIATING RECOVERY"
fi

Recovery Protocol

If integrity check fails after a stage:

1. Stop immediately — do not continue to next stage
2. Locate most recent backup:

   ls -td ${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-* | head -1
   

3. Restore from backup:

   BACKUP=$(ls -td ${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-* | head -1)
   rm -rf "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"  # Remove corrupted workspace
   cp -r "${BACKUP}" "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"
   echo "Restored from ${BACKUP}"
   

4. Mark current stage as failed in state.json
5. Report the failure with:
   • Which stage was running
   • What integrity check failed
   • Which backup was restored
   • Any error output captured

Subagent Sandboxing

When spawning subagents via pi -p --no-session, the orchestrator MUST prepend these safety instructions to the prompt:

CRITICAL SAFETY CONSTRAINTS:
1. DO NOT delete the workspace directory (${LYCEUM_TEXTS_DIR:-output/texts}/SLUG)
2. DO NOT run `rm -rf` on any parent directory of the workspace
3. DO NOT run destructive commands outside the workspace
4. ONLY modify files WITHIN the workspace directory
5. If you encounter an error, report it - do not attempt cleanup by deletion
6. The workspace directory MUST still exist when you complete

Workspace path: ${LYCEUM_TEXTS_DIR:-output/texts}/SLUG
This directory and its contents must remain intact.

Timeout Configuration

Default timeouts by stage complexity:

Stage Type	Timeout	Notes
Simple (intake, transliteration)	300s	Minimal external dependencies
Medium (cleaning, segmentation)	600s	File processing
Complex (source-hunt, versification)	900s	May involve web lookups or LLM calls
Very Complex (interlinear)	1800s	Minimum for small texts (<500 words). Use 3600s for texts >1000 words in agent mode

Stage 8 timeout guidance: In agent mode, Stage 8 requires ~5 minutes per 100 words due to subprocess overhead. A 1600-word text takes ~90 minutes. Set timeout to at least 1800s for small texts, and 3600s (1 hour) for larger works. API mode is 5-10x faster and can use shorter timeouts.

Subagent Invocation Pattern

The safe pattern for invoking stage skills:

SLUG="your-text-slug"
STAGE="N-stage-name"
SKILL="skill-name"
WORKSPACE="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}"
TIMEOUT=900

# 1. Create backup
BACKUP="${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-${STAGE}-$(date +%Y%m%d-%H%M%S)"
cp -r "${WORKSPACE}" "${BACKUP}"

# 2. Spawn subagent with safety constraints and timeout
timeout ${TIMEOUT} pi -p --no-session "
CRITICAL SAFETY CONSTRAINTS:
- DO NOT delete ${WORKSPACE}
- DO NOT run rm -rf on parent directories
- Workspace MUST exist when you complete

/skill:${SKILL} run ${SLUG}
" 2>&1 | tee "${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}/logs/${STAGE}.log"

SUBAGENT_EXIT=$?

# 3. Verify integrity regardless of exit code
if [[ ! -d "${WORKSPACE}" ]] || [[ ! -f "${WORKSPACE}/manifest.json" ]] || [[ ! -f "${WORKSPACE}/state.json" ]]; then
  echo "INTEGRITY CHECK FAILED"
  echo "Restoring from backup: ${BACKUP}"
  rm -rf "${WORKSPACE}" 2>/dev/null
  cp -r "${BACKUP}" "${WORKSPACE}"
  # Mark stage failed
  echo "Stage ${STAGE} failed: workspace integrity violated"
  exit 1
fi

# 4. Check subagent exit status
if [[ ${SUBAGENT_EXIT} -eq 124 ]]; then
  echo "WARNING: Subagent timed out after ${TIMEOUT}s"
  # Workspace intact, but check if stage completed
fi

# 5. Clean up old backups (keep last 3)
ls -td ${LYCEUM_TEXTS_DIR:-output/texts}/${SLUG}.backup-* 2>/dev/null | tail -n +4 | xargs rm -rf 2>/dev/null

Orchestrator Loop with Defenses

Updated stage execution loop:

FOR each stage in pipeline:
  1. ANNOUNCE: "=== Pre-Stage ${stage}: Creating Backup ==="
  2. CREATE BACKUP of workspace
  3. VERIFY backup exists
  
  4. ANNOUNCE: "=== Executing Stage: ${stage} via ${skill} ==="
  5. SPAWN subagent with:
     - Safety constraints prepended to prompt
     - Appropriate timeout
     - Output logging
  
  6. ANNOUNCE: "=== Post-Stage ${stage}: Integrity Check ==="
  7. VERIFY workspace integrity:
     - Directory exists
     - manifest.json valid
     - state.json valid
  
  8. IF integrity check FAILED:
     - RESTORE from backup
     - MARK stage as failed
     - STOP orchestration
     - REPORT failure details
  
  9. IF stage completed successfully:
     - RUN VERIFICATION SCRIPT (see below)
     - IF verification FAILS: MARK stage blocked, STOP
     - IF verification WARNS: record warnings, ADVANCE
     - IF verification PASSES: ADVANCE to next stage
  
  10. CLEANUP old backups (keep last 3)

---

Verification Scripts

CRITICAL: After every stage completes, the orchestrator MUST run the corresponding verification script before advancing. These scripts verify quality, not just file existence.

Stage-to-Script Mapping

Stage	Script	What it verifies
0-intake	scripts/verify_stage_0.sh	URNs, manifest, directories
1-source-discovery	scripts/verify_stage_1.sh	Source provenance, recommendations
2-acquisition	scripts/verify_stage_2.sh	Greek/English content, file sizes
3-cleaning	scripts/verify_stage_3.sh	No HTML, no bleed-through, report quality
4-segmentation	scripts/verify_stage_4.sh	Segment counts, DB import, cross-check
5-witnesses	scripts/verify_stage_5.sh	Catalog roles, provenance, file existence
6a-translation	scripts/verify_stage_6a.sh	Coverage, reviews, metadata, DB cross-check
6b-versification	scripts/verify_stage_6b.sh	Generated edition, segment match, no empties
7-transliteration	scripts/verify_stage_7.sh	Artifact existence, no Greek leakage
8-interlinear	scripts/verify_stage_8.sh	Gloss quality, completeness, dictionary-style detection, ground-truth benchmark presence
9-reader-reliability	scripts/verify_stage_9.sh	Full reader stack, feature matrix, server build
10-ship-gate	scripts/verify_stage_10.sh	Runs ALL prior verifications, final cross-check

Invocation Pattern

SLUG="your-text-slug"
STAGE_SCRIPT="scripts/verify_stage_N.sh"  # or 6a, 6b, etc.

bash "${STAGE_SCRIPT}" "${SLUG}"
VERIFY_EXIT=$?

case $VERIFY_EXIT in
  0) echo "VERIFICATION PASSED" ;;
  1) echo "VERIFICATION FAILED — blocking stage advancement"
     # Mark stage as blocked with verification failure
     ;;
  2) echo "VERIFICATION WARNINGS — proceeding with notes"
     # Record warnings in stage history
     ;;
esac

Exit Code Semantics

Exit Code	Meaning	Action
0	PASS — all checks passed	Advance to next stage
1	FAIL — critical check failed	Block stage, stop orchestration
2	WARN — non-critical issues	Advance with warning notes in history

Special Cases

Stage 6 verification depends on the path taken:

• Fast path (witness versification succeeded): run only verify_stage_6b.sh (the versified witness satisfies the same structural contract)
• Standard path (generation required): run verify_stage_6a.sh after translation-synthesis, then verify_stage_6b.sh after versify

Stage 8 verifies gloss quality, not just existence:

• Contextual glosses (not dictionary-style comma lists)
• No Greek characters in English glosses
• Lemma and transliteration completeness
• Workspace ground-truth benchmark/report artifacts exist before promotion
• Benchmark status in qa/interlinear-report.md is not not_available
• Samples the first segment's word table for human review

Stage 9 simulates the reader view:

• Assembles what the /read/ handler would show for segment 1
• Verifies all reader features are available (row view, glosses, popup, sidebar)
• Prints a feature support matrix

Stage 10 is a full cross-check:

• Runs ALL prior stage verifications end-to-end
• Reports a final SHIP / WARN / NOT READY verdict

---

Example Run Transcript

User: /text-orchestrator run "Plato Allegory of the Cave"

Orchestrator: === Initializing Pipeline ===
Loading skill: text-intake
Executing: /text-intake init "Plato Allegory of the Cave" --mode add --profile add

[text-intake runs, creates workspace]

Workspace created: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-allegory-of-the-cave
Mode: add
Profile: add
Requested stages: 0-intake through 10-human-review

=== Stage 1: Source Discovery ===
Loading skill: source-hunt
Executing: /source-hunt run "plato-allegory-of-the-cave"

[source-hunt discovers Greek sources from Perseus, English from Jowett]

Stage 1 complete. Found:
- Greek: Perseus TEI (recommended)
- English: Jowett 1892 (recommended)

=== Stage 2: Acquisition and Extraction ===
Loading skill: source-extract
Executing: /source-extract run "plato-allegory-of-the-cave"

[... continues through stages ...]

=== Stage 10: Human Review ===
Pipeline reached the ship gate.
Automatic execution complete.

Review pack available at: ${LYCEUM_TEXTS_DIR:-output/texts}/plato-allegory-of-the-cave/qa/final-review-pack.md

To approve and ship:
  /new-text-ship promote "plato-allegory-of-the-cave"

Human approval required. Orchestrator stopping.

---

Key Files

File	Purpose
internal/textpipeline/textpipeline.go	Stage definitions and state model
scripts/init_text_pipeline_workspace.go	Workspace initializer
docs/text-pipeline-master-plan-2026-03-13.md	Canonical stage model
docs/text-pipeline-skill-architecture-2026-03-13.md	Skill ownership
.pi/skills/*/SKILL.md	Individual stage skills

---

Implementation Notes

This Is an Agent Skill, Not a Script

The orchestrator works by:

1. Reading this skill file to understand the protocol
2. Sequentially loading and executing stage skills
3. Checking state between stages
4. Making decisions about continuation

The "automation" comes from the agent following these instructions without waiting for human prompts between stages.

Subagent Invocation

When the orchestrator "invokes" a skill, it means:

1. Create backup of the workspace FIRST
2. Read the skill's SKILL.md file
3. Prepend safety constraints to the invocation prompt
4. Execute the skill's run command with appropriate timeout
5. Verify workspace integrity immediately after completion
6. If integrity check fails, restore from backup and stop
7. Follow the skill's instructions
8. Return to the orchestrator protocol

This is effectively the same agent wearing different "hats" for each stage.

Known Gotcha: nix-shell Banner Pollution

Do NOT pipe nix-shell -p <pkg> --run "command" output into data files or DB imports.

The project's shellHook prints a banner to stdout that contaminates captured output. Use jq directly (available in the dev shell) or redirect stderr.

Fixed: The flake.nix shellHook now only prints in interactive shells, but avoid the pattern anyway. Use tools from the dev shell directly, or pipe to files if needed.

CRITICAL: Subagent Trust Model

Subagents are NOT sandboxed by the system. The allowed-tools field in skill files is documentation only — it is NOT enforced. Subagents spawned via pi -p --no-session have full unrestricted tool access.

This means:

• A subagent CAN delete the entire workspace
• A subagent CAN run arbitrary destructive commands
• A subagent CAN corrupt state files

The orchestrator's defenses (backup, integrity check, recovery) exist because we CANNOT trust subagent behavior. Always:

1. Backup before spawning
2. Verify after completion
3. Be prepared to restore and abort

Parallelization

The current model is strictly sequential. Future enhancements could allow parallel execution of independent stages, but the dependency graph (see invalidationRules in textpipeline.go) must be respected.