Setup indexing

Setup code indexing for agents

Install a per-repo greppable index at .claude/index/<member>.md with three sections — ## dirs, ## files, ## symbols (path:line:definition) — so an agent answers "where is X?" with one grep on one file instead of fanning Glob/Grep/Read across the tree. Measured effect on a real multi-repo workspace: same accuracy as free exploration, ~30% less context processed, ~40% less wall time, zero junk matches (worktree/vendored copies excluded by construction).

The work has two halves and the second is not optional: install, then validate with measurements. Every gate in the validation pipeline exists because skipping it shipped a real bug at least once.

1. Survey the project (before writing anything)

• Layout: single repo, or a workspace whose members are separate gitignored clones (check .gitignore for directory entries like /app/)? Each indexed unit needs its own .git.
• Submodules: cat .gitmodules in each member. git ls-files does not descend into them — the template handles this, but you need to know they exist to verify they got indexed.
• Languages: the template covers TS/JS (incl. routes), Rust, Go, shell, Python, and extensionless node/sh/python shebang executables. Anything else (Ruby, Java, C#…) needs a PAT_* + an all_symbols line — POSIX ERE only (must run on BSD grep), anchor definitions, favor recall over precision.
• Existing tooling: if the project already has ctags/LSP index generation, prefer extending that over installing a parallel system — surface the choice to the user.

2. Install

1. Copy assets/index.sh.template to the project root as index.sh, chmod +x. Set MEMBERS_ALL: "root" alone for a single repo; append member dir names for a workspace ("root app shared hub"). Adapt the header comment to the project's voice.
2. Add /.claude/index/ to .gitignore before the first run — otherwise the root index lists itself and every regeneration dirties the tree.
3. Run ./index.sh. Read one generated file's header and spot-check a few symbol lines before proceeding.

Why the template is the way it is (don't simplify these away):

• Targets stock macOS bash 3.2 + BSD tools under set -euo pipefail. The || true on grep pipelines is load-bearing: grep exits 1 on no-match, which under pipefail truncates the index mid-write and leaves a header that makes --check lie "fresh".
• git ls-files (tracked + --others --exclude-standard) instead of find/rg: excludes worktrees, node_modules, build output by construction — this is where the measured quality win comes from.
• Header carries HEAD and a working-tree fingerprint: HEAD alone misses uncommitted edits, so line numbers drift while --check says fresh.
• grep -e "$pat" --: a dash-prefixed filename otherwise becomes a grep option and silently empties the whole batch.
• sort -u | cut -c1-250: embedded-blob consts (multi-KB single lines) would drown every grep that touches them.

3. Wire it up

1. CLAUDE.md (project root) — add a section so every session learns the index exists; without this the index is dead weight. Pattern:

   ## Code index — locate things WITHOUT scanning the repos
   
   `.claude/index/<member>.md` (gitignored, generated by `./index.sh`) is a
   greppable index: `## dirs`, `## files`, `## symbols` (path:line:definition).
   **Before reaching for Glob/Grep across the tree, grep the index first:**
   
   ​```bash
   grep -n "SomeSymbol" .claude/index/<member>.md   # symbol → file:line
   ./index.sh --check || ./index.sh                  # refresh if stale (seconds)
   ​```
   

2. Member CLAUDE.md files (multi-repo only): a short conditional note — "when this clone lives inside the <workspace> dev workspace, a greppable index of this repo is at ../.claude/index/<m>.md …". Conditional phrasing matters: standalone clones must not be misled. One commit per member repo; ask before pushing.

3. SessionStart hook (optional, recommended): auto-refresh at session start. Do not edit .claude/settings.json yourself — permission classifiers (rightly) block agents self-modifying startup config; it was denied when attempted. Give the user the JSON and let them add it:

   {
     "hooks": {
       "SessionStart": [
         { "hooks": [ {
             "type": "command",
             "command": "\"$CLAUDE_PROJECT_DIR\"/index.sh --quiet 2>/dev/null || true",
             "async": true,
             "timeout": 120,
             "statusMessage": "Refreshing code indexes"
         } ] }
       ]
     }
   }
   

   async: true so a 2-second regeneration never blocks session start; the previous index stays usable meanwhile. Without the hook the setup still works — the CLAUDE.md instruction makes agents run --check themselves.

4. Validate

Read references/validation.md and run the gates: (1) syntax + full run, (2) edge-case sandbox (empty/flat/dash-file repos), (3) measured recall check — 8 elements picked by independent exploration, ≥7/8 locatable via the index alone, any fully-missing category is a fail regardless of score, (4) staleness detection (uncommitted-edit fingerprint), (5) optional A/B benchmark when the user wants numbers.

Gate 3 is the heart. The first build of this system scored 8/8 on four repos and 4/8 on the fifth — every miss was a category (non-exported JS defs, extensionless node executables) invisible to the initial patterns. Only the measurement caught it.

5. Report

Summarize: files installed, index sizes + generation time, validation table (gate / pass / one-line evidence), what was skipped, and the hook JSON if the user hasn't applied it yet. Remind: regenerate after big refactors; --check is cheap (~0.1s) and catches drift including uncommitted edits.