Wiki build

/wiki-build — text_corpus → Karpathy wiki

Purpose

Build a navigable markdown wiki for future LLM use.

This skill starts AFTER corpus preparation.

Input contract:

text_corpus/        # markdown/text/transcripts/OCR outputs
inventory/          # source manifests and conversion reports
raw/                # original source layer, read-only reference

Output:

knowledge/<topic>/
  INDEX.md
  _sources.md
  _log.md
  contradictions.md
  source_notes/
    README.md
    <bucket>/<source_id>_<slug>.md
  NN_<topic_node>.md

Two semantic layers (mandatory, not optional):

1. Source notes (source_notes/<bucket>/*.md) — digested per-source memory cards. One per significant source or duplicate group. Frontmatter, 5-15 key ideas, verbatim phrases, feeds-into hints, caveats. Replaces routing-only files like _pdf_sources.md (those become audit-only).
2. Topic nodes (NN_*.md) — synthesized memory by topic. Reference source notes via [[source_notes/<bucket>/<id>#anchor]] evidence-links, not raw file paths.

Routing/provenance files (_pdf_sources.md, _image_sources.md, etc.) may exist for audit but are NOT semantic memory. Topic nodes must not point agents to raw parsed files for first-pass comprehension.

If text_corpus/ does not exist or raw files are still unconverted, call/propose /corpus-ingest first. Do not silently parse a messy raw dump inside this skill.

When to use

Triggers:

• "Build a knowledge base on topic X"
• "Lay out the course/corpus into a playbook"
• "Make a wiki on <domain> / ML papers / medicine"
• "Make a Karpathy wiki from these markdown/transcripts/OCR files"
• Corpus already has text_corpus/ and inventory/

Do NOT use directly:

• raw archives / Telegram export / cloud dump / media dump → /corpus-ingest
• one tiny document → read directly and answer
• repetitive per-record extraction → /batch-analyze as internal utility
• health check of existing wiki → /wiki-lint

Karpathy principle

3 layers:

1. Sources — raw originals + _sources.md, immutable.
2. Wiki — markdown nodes the LLM can read and maintain.
3. Project routing — INDEX.md / CLAUDE.md / memory pointers.

The LLM should read INDEX.md first, then only 1-3 relevant nodes. The wiki is a context router, not a dump of source text.

Phase 0 — Preflight

1. Locate project and topic path.
2. Verify input contract:
   • text_corpus/ exists
   • inventory/ exists
   • source/conversion reports are available or explain missing parts
3. Estimate corpus size and modality mix.
4. Identify gaps from ingest reports: empty PDFs, skipped OCR, partial cloud, etc.
5. Present plan and STOP for confirmation.

Phase 1 — Corpus map

Build or update:

knowledge/<topic>/_sources.md
knowledge/<topic>/_log.md
knowledge/<topic>/TOPIC_PLAN_DRAFT.md

Source map must reference normalized text outputs and original raw paths where possible.

For each source bucket, record:

• source ID
• normalized file path
• original/raw path
• type
• status/quality
• caveats

Phase 2 — Topic plan

Read/summarize representative sources and create 5-20 top-level nodes.

Rules:

• one node = one durable topic
• node name must be useful for future routing
• do not create 50+ flat nodes; use subfolders if needed
• include caveats for incomplete/low-quality corpus parts

Show the topic plan to the user and STOP before writing the final wiki if the map is non-trivial.

Phase 2.5 — Source notes (semantic digest)

Mandatory before topic-node synthesis if the corpus has more than ~10 significant sources.

For each significant source (or duplicate group), create one source note in source_notes/<bucket>/:

---
source_id: <NNNN>
source_path: text_corpus/<...>.md
raw_origin: raw/<...>.<ext>
authority: <school/author/program>
date_origin: YYYY-MM-DD or unknown
quality: useful_text | low_text | empty_or_scan
chars: <int>
duplicates_of: [NNNN, NNNN]
feeds_topics: [NN, NN]
updated: YYYY-MM-DD
---

# Source title

## What is this
1-2 lines.

## Key ideas / rules
- 5-15 bullets, concrete rules with section/page refs.

## Concrete phrases / scripts
- 3-10 verbatim phrases when source is a script.

## Feeds into
- NN#anchor — what flows where
- NN#anchor

## Caveats
- ...

## Already synthesized
_pending_  (filled when topic nodes are written)

Rules:

• one note = one significant source (group duplicates by content hash, list duplicates_of)
• 80-250 lines max — never copy the whole parsed source
• use stable anchors (## priem-5) so topic nodes can deep-link
• if source is empty/scan, create a 10-line stub explaining what is missing
• prioritize useful_text high-authority sources first; low-text and noisy can wait

Source notes go into knowledge/<topic>/source_notes/<bucket>/ (bucket: pdf, transcripts, images, cloud, tg etc.) with a source_notes/README.md explaining the layer.

Routing-only files (_pdf_sources.md) are kept for audit but renamed/marked as audit-only. They MUST NOT be cited from topic nodes as the route to comprehension.

Show the first 1-2 example source notes to the user and STOP for shape approval before batch-generating the rest.

For batches >20 sources, use /batch-analyze as an internal utility to parallelize digestion. Validate a sample manually.

Reference example (REQUIRED reading before generating)

A reference source note that should be used as the shape baseline:

knowledge/<topic>/source_notes/pdf/0062_example.md

Why it is the reference:

• frontmatter has every field (source_id, source_path, raw_origin, authority, date_origin, quality, chars, duplicates_of, feeds_topics, updated)
• multiple numbered concrete rules with stable anchors (### priem-1 ... ### priem-33)
• bold-led one-line claim per rule, then 1-2 lines of supporting detail
• separate ## Concrete phrases/scripts block with verbatim scripts
• ## Feeds into lists target topic#anchor, not just topic numbers
• caveats include source-school context, not just "text is partial"

Canonical subagent prompt for batch source-note digestion

Use this verbatim when launching a subagent to generate N source notes from parsed corpus. Substitute {ITEMS}, {REFERENCE_PATH}, {OUTPUT_DIR}, {TOPIC_MAP_HINT}.

Task: create {N} source_notes files for the {project} wiki using the Karpathy semantic-layer method.

Context and format — read the template `{path}/knowledge/<topic>/source_notes/README.md` and the reference source note `{REFERENCE_PATH}`. Their structure is mandatory.

Create N files in `{OUTPUT_DIR}`:

{ITEMS — list of the form: filename ← source_path (short_title, size, duplicates, feeds_topics)}

REQUIREMENTS for each source_note:
- Frontmatter exactly as in the README template (all fields)
- **Key ideas/rules**: 5-15 bullets that capture the essence of the source faithfully. No fluff paraphrasing — concrete rules, numbers, steps, formulations. Every bullet must be worth a place in the agent's memory
- Each bullet starts with a **bold-claim** on one line, then 1-2 lines of detail
- A stable anchor (`### priem-5`, `### oshibka-7`, `### vozr-name`) before each bullet for evidence-links from topic nodes
- **Concrete phrases/scripts**: 3-10 verbatim scripts/formulations from the source (unmodified). If the source is a script with a list of objections — add an anchor to each one (`### vozr-dorogo`)
- **Feeds into**: explicitly state which sections of the topic nodes get what (format: `05#price-objections — techniques 1-15`, not just `05`)
- **Caveats**: school/authority context, outdated, needs adaptation, duplicate, scan-OCR-noise
- **Already synthesized**: leave `_pending wiki-build/NN sync_` — filled in when topic nodes are synthesized
- Length: 80-250 lines. No more — the wiki must not become a copy of the source
- Language: match the corpus in the body, English keys in frontmatter
- Do NOT copy the whole source — only a digested distillation

TOOLS: Read to parse the sources, Write to create source_notes. You may do parallel reads.

IMPORTANT: after creating all files — report one line per file: "{source_id} OK, NN rules, NN scripts" — for quick validation.

If you find clear contradictions with other sources in the corpus (a different sales school, opposite advice, conflicting numbers) — do NOT stay silent; in a separate block at the end of the report, list "Counter-found: X vs Y on topic Z, recommend adding to contradictions.md".

Do not create README.md, INDEX.md, or contradictions.md, and do not touch topic nodes — only source_notes/<bucket>/*.md.

Batch sizing rules

• 6-10 sources per subagent call. Fewer — overhead. More — the report is less reproducible.
• Run 2-4 subagents in parallel (background) for different thematic clusters.
• Do NOT run one file at a time — it is 5x more expensive and 3x slower.
• After the batch — spot-check 1 random file manually (frontmatter + anchors + length).

Quality gate per source note

Before commit, check deterministically:

• frontmatter complete (all 10 fields)
• at least 5 anchors ^###
• length 60-260 lines
• section ## Feeds into exists and contains NN#anchor style
• section ## Concrete phrases/scripts exists if the source is a script
• no markdown duplicates # title # title

Phase 3 — Node creation

For each node:

---
topic: <topic_slug>
node: NN_<node>.md
source_refs: S001, S002
confidence: HIGH|MEDIUM|LOW
updated: YYYY-MM-DD
---

# Node title

For future Claude: 2-3 lines explaining when to read this node.

## Key ideas
...

## Concrete scripts/examples/facts
...

## Caveats
...

## See also
- [other_node.md](other_node.md)

Quality rules:

• synthesize, do not copy large source text
• include concrete phrases/examples only when useful
• keep nodes around 50-200 lines unless the topic demands more
• cite source notes via evidence-links: [[source_notes/<bucket>/<id>#anchor]], not raw text_corpus/ paths
• never tell the agent "see these PDFs" as the answer — that is a routing failure
• mark confidence and coverage limitations
• record cross-source disagreements in contradictions.md instead of picking silently

Reference example for topic-node rewrite (REQUIRED)

knowledge/<topic>/05_example_section.md and 06_example_section.md — canonical shape after evidence-link rewrite.

What to copy:

• Frontmatter source_refs: lists [[source_notes/...]] IDs only, no raw paths
• Each major claim ends with → [[source_notes/<bucket>/<id>#anchor]]
• Section anchors at level ## Section name {#anchor} so other nodes can link
• ## Coverage caveats lists what is NOT yet digested (specific source_ids), with a pointer to _pdf_sources.md as audit-only
• ## Contradictions section, even if "no open forks"
• ## See also cross-refs all related topic nodes

Phase 4 — INDEX.md

INDEX.md is the router. Route by intent, not by topic name.

Weak (rejected):

- Need objections → 06_objections.md

Required:

- Owner says "come only with a buyer" → 06_objections.md#seller-only-with-buyer + 07_owner_acquisition.md
- Buyer says "too expensive" → 10_negotiation.md#price-objection + [[source_notes/pdf/0062_example#priem-1]]
- Agent discounts too early → 10_negotiation.md#early-discounting

Must include:

• what is inside (incl. mention of two semantic layers)
• intent-level routes (grouped by user-facing situation, not by file name)
• all topic nodes with one-line purpose
• pointer to source_notes/ and explanation that _pdf_sources.md/etc. are audit-only
• pointer to contradictions.md
• sources link, log link, coverage caveats

Phase 5 — Validation before final answer

Minimum checks:

• all markdown links resolve (incl. [[source_notes/...]] evidence-links)
• every node is referenced from INDEX.md
• INDEX has intent-level routes, not only topic listings
• every significant source has either a source note or an explicit caveat in 16_assets_unprocessed_materials.md (or equivalent)
• topic nodes do NOT cite raw text_corpus/ paths as the route to comprehension
• contradictions.md exists (even if empty)
• _sources.md exists and points to real files/reports
• _log.md has a current entry
• no obvious duplicate node names
• coverage caveats match ingest reports

For large corpora, /batch-analyze may be used internally for topic clustering or extract/verify passes. It remains a utility, not a wiki stage.

Phase 6 — Memory and report

After completion:

• add/update one line in your project's MEMORY.md pointing to knowledge/<topic>/INDEX.md
• append a daily note with path, node count, source count, caveats
• report what is done and what remains outside the wiki

Anti-patterns

• Do not run raw ingestion here when /corpus-ingest is needed.
• Do not create a wiki from unconverted PDFs/media and pretend coverage is complete.
• Do not paste huge source excerpts into nodes.
• Do not skip the topic-plan checkpoint for large corpora.
• Do not skip Phase 2.5 source notes for corpora >10 significant sources — that produces a routing-only wiki, not semantic memory.
• Do not let topic nodes cite raw text_corpus/ paths as the route to comprehension. Use [[source_notes/...]] evidence-links.
• Do not pretend _pdf_sources.md (or similar provenance maps) is wiki memory. It is audit metadata.
• Do not silently choose between contradictory sources — record in contradictions.md.
• Do not forget coverage caveats.
• Do not let /batch-analyze become a second wiki builder.