Skill 0 ingest

Skill 0: Ingest Docs

<SUBAGENT-STOP> If you were dispatched as a subagent to execute a specific task, skip this skill. </SUBAGENT-STOP>

Convert multiple human-authored Markdown files into the YAML artifacts that skill-3-task-gen and the rest of the prd2impl pipeline expect. This is Entry B — an alternative to running /prd-analyze (Entry A) when the project already has hand-written gap analyses, design specs, or plans.

Trigger

• User runs /ingest-docs [file1.md] [file2.md] ...
• User provides --tag role=path overrides alongside file paths
• User provides --synthesize-user-stories flag (opt-in LLM synthesis of user_stories for design-spec role — see §Inputs)
• User says "ingest my docs", "read my markdown files", "import gap analysis and design spec"

Inputs

• Required: One or more MD file paths

• Optional: --tag <role>=<path> overrides (force a role instead of auto-detecting)

  • Valid roles: gap, design-spec, plan, prd, user-stories
  • Example: --tag design-spec=a.md --tag gap=b.md

• Optional: --synthesize-user-stories — opt-in flag. When passed and any input file is classified as role=design-spec, run the LLM synthesis pass described in lib/prd-extractor.md §user_stories LLM synthesis. Default: off, which preserves existing user_stories: [] behavior for design-spec.

Outputs

Three YAML files written to {plans_dir}/{date}-*.yaml:

File	Produced when	Consumed by
{date}-prd-structure.yaml	Any prd, plan, user-stories, or design-spec MD found	skill-3-task-gen
{date}-gap-analysis.yaml	Any gap MD found	skill-3-task-gen
{date}-task-hints.yaml	Any design-spec or plan MD found	skill-3-task-gen (optional)

All files carry source_type: "ingested" to distinguish them from skill-1/skill-2 outputs.

---

Phase 1 · Role Detection

Read: lib/role-detector.md — follow it exactly for this phase.

Steps:

1. For each input file, run the 4-signal heuristic scorer.
2. For any file scoring < 70, invoke LLM fallback (see role-detector §LLM fallback).
3. Apply any --tag overrides (force confidence=100 for those files).
4. Build the confirmation table using templates/role-confirmation.md.
5. Print the table to the user.

[HUMAN REVIEW CHECKPOINT 1]

Present the table and wait. Accept commands:

• ok → proceed to Phase 2
• override N <role> → update role for row N; re-display table; ask again
• drop N → remove file from processing list; re-display; ask again

If all remaining files are unknown → abort:

ERROR: All input files classified as 'unknown'. Re-run with --tag role=path to specify roles.

---

Phase 2 · Extraction (per-role, sequential)

Important: Run extractors in the order: gap → spec → prd. This order ensures gap_analysis is populated before spec-extractor's cross-references need it.

2a. Gap extraction

If any files have detected_role: gap:

• Read: lib/gap-extractor.md — follow it exactly.
• Build gap_analysis dict in memory.
• Print: Extracted {N} gaps (P0={n0}, P1={n1}, P2={n2}) from {files}

2b. Spec extraction

If any files have detected_role: design-spec:

• Read: lib/spec-extractor.md — follow it exactly for task-hints extraction.
• Read: lib/prd-extractor.md §Extraction: role=design-spec — follow it for modules/nfrs/constraints extraction.
• Build task_hints dict in memory (from spec-extractor).
• Build or extend prd_structure dict in memory (from prd-extractor, partial: modules/nfrs/constraints only).
• Print: Extracted {F} file_changes, {S} steps, {G} non_goals (spec) + {M} modules, {N} nfrs, {C} constraints (design) from {files}

If any files have detected_role: plan:

• Read: lib/spec-extractor.md — Phase 0 detects writing-plans format and delegates to lib/plan-parser.md. For writing-plans-format md (role-detector Signal 0 hit), the output is task_hints.tasks[] (Task → Files → Steps hierarchy preserved verbatim). For legacy plans (no writing-plans header), the legacy step-extraction flow (Steps 1-7) runs.
• Read: lib/prd-extractor.md §Extraction: role=plan for module extraction.
• For each entry in task_hints.tasks[] (when present), populate source_plan_path with the input file's repo-relative path. The parser leaves this null; only the caller knows the path.
• Merge step data into task_hints; merge module data into prd_structure.
• Print, depending on path:
  • writing-plans format: Extracted {T} tasks ({S} total steps) (plan-passthrough) + {M} modules from {files}
  • legacy format: Extracted {S} steps (plan-legacy) + {M} modules from {files}

2c. PRD / user-stories extraction

If any files have detected_role: prd or detected_role: user-stories:

• Read: lib/prd-extractor.md — follow the appropriate role routing.
• Build or extend prd_structure dict in memory.
• Print: Extracted {M} modules, {U} user stories, {N} NFRs from {files}

---

Phase 3 · Cross-Validation

Read: lib/cross-validator.md — follow it exactly for this phase.

Steps:

1. Run Rule 4 (fatal check) first.
   • If any fatal: print conflict details; abort — do not write any files.
2. Run Rules 1, 2, 3, 5 (warning checks). Collect all warnings.
3. If any warnings: print warning table.

[HUMAN REVIEW CHECKPOINT 2] (only if warnings exist)

Cross-validation complete. {F} fatal errors, {W} warnings.

{warning_table}

Proceed despite warnings? (y/n):

• n → abort (no files written)
• y → proceed to Phase 4

If no warnings (and no fatals): skip checkpoint, proceed automatically.

---

Phase 4 · Write + Summary

Path resolution: Before constructing any output path, resolve {plans_dir} per lib/plans-dir-resolver.md. All paths below use {plans_dir} as the base directory.

4.0 Extraction Completeness Check

Before writing output files, verify extraction completeness against source documents:

1. Collect source item counts — prefer using the per-role extraction counts already reported by Phase 2:

   • Phase 2a (gap) already prints: Extracted {N} gaps (P0={n0}, P1={n1}, P2={n2}) from {files}. Use {N} as the extracted gap count.
   • Phase 2b (spec/plan) already prints extraction counts for file_changes, steps, and non_goals.
   • Phase 2c (prd) already prints extraction counts for modules and user stories.
   • Only re-read source files to get raw item counts if the Phase 2 output lines did not capture a specific category.

2. Compare extraction counts against what ended up in the in-memory YAML dicts:

   • For design-spec and legacy plan: task_hints.file_changes count vs Phase 2b reported file_changes count
   • For writing-plans-format plan (plan-passthrough): if task_hints.tasks is non-empty, use len(tasks) vs Phase 2b reported task count; skip the file_changes comparison
   • gap_analysis.gaps count vs Phase 2a reported gap count
   • prd_structure.modules count vs Phase 2c reported module count

3. If any ratio is < 70% (and source count ≥ 3 to avoid false positives on tiny docs), print a WARNING:

   ⚠️  Extraction completeness warning:
     file_changes: 6 extracted from 16 detected in source (37%) — design-spec may have non-standard formatting
     modules: 4 extracted from 5 detected in source (80%)
     Consider manual review of the output YAMLs.
   

4. If all ratios ≥ 70%: print nothing (clean extraction, no noise).

5. This check is INFORMATIONAL only — never blocks the pipeline.

4.1 Determine output paths

{plans_dir}/{date}-gap-analysis.yaml      (if gap_analysis populated)
{plans_dir}/{date}-prd-structure.yaml     (if prd_structure populated)
{plans_dir}/{date}-task-hints.yaml        (if task_hints populated)

Where {date} = today's date in YYYY-MM-DD format.

If a file already exists at the target path: print a diff-summary line (informational, non-blocking) and overwrite.

Required diff-summary format (enforced — tests match on this):

Overwriting {path} ({old_count} → {new_count} {entity}[, {old_N} → {new_N} {entity2}]...)

• MUST start with the literal word Overwriting followed by the absolute/relative path.
• MUST contain at least one {old_count} → {new_count} {entity} delta inside parentheses, using the → arrow character (U+2192).
• Per-artifact primary entities (use these labels; add secondary deltas as relevant):
  • gap-analysis.yaml → gaps (primary), P0 / P1 / P2 (secondary)
  • prd-structure.yaml → modules (primary), user_stories / constraints (secondary)
  • task-hints.yaml → file_changes or tasks (primary, whichever is non-empty), steps / non_goals (secondary). When source role is plan AND writing-plans format (plan-passthrough path), tasks count is primary; the secondary delta is the sum of len(t.steps) for t in tasks[].
• If old_count is 0 (file did not exist before overwrite — shouldn't reach this branch, but defensive), skip the diff and print Writing {path} ({new_count} {entity}) instead.

Required downstream-invalidation hint (print verbatim after the overwrite):

Note: tasks.yaml for this plans_dir already exists and may now be inconsistent. Re-run /task-gen to regenerate.

Only print the hint if {plans_dir}/tasks.yaml exists at overwrite time; otherwise skip it (no downstream to invalidate yet).

Users who want to keep both versions should run with a different --plans-dir.

Worked example — re-running ingest on an updated PRD:

Overwriting docs/plans/m2/2026-04-20-prd-structure.yaml (12 → 13 modules, 47 → 49 user_stories)
Overwriting docs/plans/m2/2026-04-20-gap-analysis.yaml (17 → 18 gaps, 6 → 7 P0)
Note: tasks.yaml for this plans_dir already exists and may now be inconsistent. Re-run /task-gen to regenerate.

4.1b Populate extraction metadata (design-spec only)

Before writing prd-structure.yaml, if prd_structure.source_role == "design-spec", add an extraction key recording which fields came from which code path:

extraction:
  regex_fields: [modules, nfrs, constraints, external_deps]
  llm_fields: []            # default

If the --synthesize-user-stories flag was passed AND the LLM synthesis actually produced ≥1 valid user story: append user_stories to llm_fields:

extraction:
  regex_fields: [modules, nfrs, constraints, external_deps]
  llm_fields: [user_stories]

If the flag was passed but the LLM pass produced 0 valid stories (trigger fired but personas all failed verbatim check, or LLM call failed), keep llm_fields: [] — downstream treats the output uniformly whether the flag was passed or not.

For source_role in {prd, plan, user-stories}: do NOT add the extraction key — all their fields come from regex, no provenance tracking needed.

Downstream skills (skill-3-task-gen, skill-12-contract-check) read prd_structure.extraction to decide where to apply fuzzy matching vs. strict validation.

4.2 Write files

Before writing each YAML file, compute and add a source_hash field:

For each source file that contributed to a YAML output, run:

git hash-object "{source_file_path}"

Add to each YAML dict before writing:

source_hash:
  path/to/source.md: "<git-hash-object-hex>"
  another/source.md: "<git-hash-object-hex>"

This applies to gap-analysis.yaml, prd-structure.yaml, and task-hints.yaml (any YAML produced from source files).

Write each populated YAML dict to its target path. Validate that:

• gap_analysis.yaml begins with gap_analysis: key
• prd-structure.yaml begins with prd_structure: key
• task-hints.yaml begins with task_hints: key

4.3 Print summary

─────────────────────────────────────────────────────
skill-0-ingest complete
─────────────────────────────────────────────────────
Input files:  {N} processed, {D} dropped, {U} unknown/skipped

Outputs written:
  {plans_dir}/{date}-gap-analysis.yaml      {N_gaps} gaps  (P0={n0}, P1={n1}, P2={n2})
  {plans_dir}/{date}-prd-structure.yaml     {N_mods} modules, {N_us} user stories
  {plans_dir}/{date}-task-hints.yaml        {N_fc} file_changes, {N_steps} steps

Cross-validation:  {W} warnings (see above), 0 fatals

─────────────────────────────────────────────────────

[HUMAN REVIEW CHECKPOINT 3]

Review the output YAMLs above. See the footer at the end of this output for next steps.

Ready to proceed? (y/n — or adjust the YAML files first)

---

Error handling reference

Scenario	Response
Input file not found	ERROR: file not found: {path} → abort before Phase 1
All files unknown after Phase 1	ERROR: all unknown → abort with --tag suggestion
One file fails extraction	Skip file + warning; continue with others
Fatal cross-validation conflict	Abort Phase 4; print conflict pairs
Same-day re-run of /ingest-docs	Overwrite with diff-summary log (no suffix; use different --plans-dir for coexistence)

Component library

All extraction logic lives in lib/. Do not duplicate logic here — read the lib file and follow it.

Phase	Read this lib file
1	lib/role-detector.md (Signal 0 = writing-plans header override → role=plan / confidence=100)
2a	lib/gap-extractor.md
2b	lib/spec-extractor.md (Phase 0 delegates writing-plans md to lib/plan-parser.md)
2b plan-passthrough	lib/plan-parser.md (used when role=plan AND writing-plans header present)
2c	lib/prd-extractor.md
3	lib/cross-validator.md

───────────────────────────────────────────────────── ⬆ /ingest-docs complete ─────────────────────────────────────────────────────

📋 Next: /task-gen — Generate tasks from ingested YAML artifacts /gap-scan — Verify codebase coverage against PRD (optional, for non-greenfield) ─────────────────────────────────────────────────────