Crux wiki

crux-wiki — a literature wiki for a crux vault

This skill adds a literature layer to a crux research vault: a living, interlinked knowledge base of prior knowledge — background, methods, SOTA, baselines, datasets, definitions — that you maintain and draw on. It is [Andrej Karpathy's LLM-wiki pattern] (compile curated sources once into a persistent wiki; keep it current; don't re-derive on every query) instantiated on crux, so crux's deterministic engine does the bookkeeping the pattern needs.

It rides on the crux skill and its engine (<crux skill>/scaffold/ — the crux skill is this skill's sibling directory in your skills dir; in a repo clone, skills/crux). If the engine is missing, install the crux skill first (npx skills add mehdiforoozandeh/crux --all, or the repo's ./install.sh). crux tracks the new knowledge you're producing (the Question→Hypothesis tree); crux-wiki tracks the existing knowledge you're building on. They are cross-linked but separate.

The one-way flow rule (hard, load-bearing)

literature (raw/) → wiki/ → informs  ask · hypothesize · interpreting findings

Never the reverse. The wiki is a literature source-of-truth. A crux project's own results — verdicts, findings, closed hypotheses — never flow into the wiki. Wiki pages may link out to nothing but other wiki pages and their raw/ sources; they must never cite a q*/h* tree node. This keeps provenance clean (every wiki claim traces to a curated source, never to your own unpublished result) and structurally prevents the self-ingestion / model-collapse failure mode that degrades agent-maintained wikis.

The reverse direction is allowed and encouraged: a tree node (a question or hypothesis) may cite [[wiki/page]] to ground itself in the literature.

The three layers

• raw/ — the PI's curated sources: papers, articles, notes, data files. Immutable. You read them; you never modify them. This is the source of truth. Only the PI decides what enters raw/ — that curation is the human's job in this pattern.
• wiki/ — the compiled pages, owned by you. Concept-slug markdown files (film-conditioning.md), plus the append-only wiki/log.md and the co-evolved wiki/SCHEMA.md. You create pages, update them as sources arrive, and maintain the links.
• wiki/SCHEMA.md — this vault's conventions (which categories it uses, page style, domain quirks). Seeded on first ingest; you and the PI co-evolve it. Read it at the start of a wiki session so you stay a disciplined maintainer, not a generic chatbot.

The engine generates one more file — WIKI.md at the vault root: the index (Karpathy's index.md), a catalog of every page by category with its one-line summary, plus the source registry with compiled status. Never hand-edit WIKI.md — it regenerates on every crux command. It is your entry point for a query.

The three operations

Ingest — crux ingest raw/<file> --title "<title>" ○→◆

The PI drops a source into raw/. You run crux ingest — the engine (○) records its sha256 in the registry, appends a ## [YYYY-MM-DD] ingest | <title> line to wiki/log.md, and re-renders WIKI.md.

Title convention: the --title carries the paper title plus the full author list and year — e.g. "Masked Autoencoders Are Scalable Vision Learners — Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick (2021)". The registry title is the only author-bearing field in the vault, so this is what makes sources findable by author (in the cockpit's wiki search, in WIKI.md, and by grep); a bare "He et al" hides every co-author. Keep it on one line (the registry and log are line-based).

Then you (◆) do the reading and synthesis the engine can't:

1. Read the source in raw/.
2. Compile it into the wiki — update existing pages where the source touches known concepts (entity/method/dataset pages), and create new pages for genuinely new concepts. A single rich source often touches many pages. Every page cites its raw/ sources in frontmatter (sources:), and every claim traces to one of them.
3. Note where the new source contradicts an existing page — flag it explicitly on the page (see the semantic-lint checklist), don't silently overwrite.
4. Run crux validate and fix any structural finding it reports.

Because the PI already curated what entered raw/, compilation is act-and-report: do it, then summarize what you filed. You do not need per-page approval.

Query — read WIKI.md → drill in → answer with citations ◆

Pure agent behavior; no engine verb. To answer a question against the wiki: read WIKI.md first to find relevant pages, open them, follow [[links]], and synthesize an answer that cites the pages and their raw/ sources. Prefer compiled wiki content over your own training knowledge.

File good answers back (Karpathy's key insight): a comparison table, a SOTA summary, a connection you discovered — if it's durable and worth keeping, save it as a new wiki page so explorations compound instead of vanishing into chat. Filed-back pages still obey the flow rule: they synthesize literature, cited to raw/ — never a project finding.

Lint — crux validate (structural) + your semantic pass ○ + ◆

Two tiers, split exactly along crux's engine/agent line:

• Structural (engine, deterministic). crux validate reports, as hard findings: broken [[wiki links]] (both directions), flow-rule violations (a wiki page citing the tree), missing title/summary, orphan pages (no inbound link), pages citing a missing source, registered-but-uncompiled sources, and source hash drift (a raw/ file's bytes changed since ingest — re-ingest it). These can't drift because the index and checks are regenerated, not hand-kept.
• Semantic (you, judgment). The engine never judges meaning; you run the health check Karpathy describes and the community's production reports converge on. Do it at the start of a wiki session — on-demand-only maintenance never happens ("vaults rot in silence"). Checklist:
  • Contradictions between pages → make it a first-class flagged state on the page (a > [!contradiction] note or a contested: true line), recording both positions with their sources and dates. Do not silently pick a winner.
  • Stale claims a newer source superseded → update the entity page in place (the page states current knowledge; the log stays the append-only history). Don't leave the old claim frozen while new facts pile up elsewhere.
  • Missing pages — an important concept cited across several sources but with no page of its own → create it.
  • Missing cross-references / thin pages — pages that should link but don't; pages too sparse to be useful.
  • Data gaps — a question the wiki can't answer yet → suggest a source for the PI to add.

Page conventions

• One thing per page — an entity, a concept, a method, a comparison, an overview. Concept-slug filenames (average-reference-baseline.md), renamed/merged freely as understanding evolves (slugs, not numbered ids — the wiki is not the tree).
• Frontmatter: type: wiki (required — this is what makes it a page), title, summary (one line — this becomes the index entry), category (entity | concept | method | comparison | dataset | overview | …), sources (comma-separated raw/… paths; every claim on the page traces to one).
• Write for the LLM reader. You are the primary reader of this wiki, not a human. Favor dense, explicit, structured prose over pretty formatting.
• Lead line extends the summary. The summary frontmatter is the index entry — don't repeat it verbatim as the page's first body line. Open with a sentence that extends it: scope, context, or the sharpest claim.
• Synthesize, don't mirror (anti-bloat). Create a page when a concept appears across ≥2 sources or is central to one — not for passing mentions. A single-source page earns its place only if it adds cross-source value — a shared metric axis, a contradiction link, a comparison caveat — beyond paraphrasing that source; if it would just restate one small source, fold the concept into a comparison or concept page instead. When your sources are short (a paragraph each), prefer 2–3 synthesis pages over one page per named method. The win is compressing facts scattered across many sources; a page that mirrors one source 1:1 adds negative value.
• Start the template from <crux skill>/scaffold/templates/wiki.md.

Setup

The wiki stands up lazily on the first crux ingest — it creates wiki/, wiki/log.md, wiki/SCHEMA.md, and the first WIKI.md. There's nothing to initialize by hand. On an existing crux vault, the first ingest bumps the engine to a wiki-aware version cleanly (old vaults keep working; the tree is untouched).

To start: point the PI at raw/ ("drop the papers/notes you want me to build on into raw/"), ingest them one at a time (Karpathy's supervised default — read the summary, guide emphasis) or in a batch, then open WIKI.md in Obsidian to see the graph.

Guardrails

• The flow rule is absolute. Literature → wiki → tree. Never file a project finding into the wiki; never let a wiki page cite a q*/h* node. crux validate enforces the second half mechanically.
• The PI curates raw/. You never decide what counts as a source. You never modify a raw/ file.
• Cite everything. Every wiki claim traces to a raw/ source. This is the guard against hallucinating into the wiki — an uncited claim is a lint smell.
• Let the engine do bookkeeping. Don't hand-maintain the index or hunt links by eye — WIKI.md is generated and crux validate finds the broken/orphan/drift cases. Your job is the reading, the synthesis, and the semantic judgments the engine can't make.
• Don't hand-edit generated content — WIKI.md, or the crux tree's META.md/ EXPERIMENTS.md. Run a crux command to regenerate.
• Optional at scale. At small scale the WIKI.md index is enough (no embedding-RAG needed). If a vault grows to many hundreds of pages, a local markdown search tool (qmd, BM25 + vector, CLI or MCP) is the natural add — but it's optional and outside this skill.

The engine (shared with crux)

crux-wiki drives the same engine the crux skill ships (<crux skill>/scaffold/): crux ingest (register a source), crux validate (tree + wiki structural lint), and crux status. Pages are plain markdown you write per the template; the engine scans them, never generates their content. Validate the install with python <crux skill>/scaffold/crux.py selftest — its # wiki layer section asserts every wiki invariant (ingest, index render, idempotency, structural lint, backward-compat) with no GPU/tokens.