Compile claims

Compile Claims + Self-Eval

Two phases: extract claims from synthesis articles, then verify a sample against raw sources.

Phase 1: Extract Claims

Read config/domain.ts for the taxonomy bucket definitions. Read the synthesis article for each bucket from wiki/{bucket-id}.md.

For each article, extract 25-40 atomic claims. Each claim is ONE verifiable statement. Not a summary, not a paragraph. One fact.

Claim Types

• empirical — numbers, benchmarks, measurements: "Mem0 selective retrieval beats full-context by 26% on LOCOMO"
• architectural — how things are built: "Graphiti stores temporal validity as edge properties in Neo4j"
• comparative — X vs Y: "Graph-based retrieval outperforms vector-only on multi-hop questions"
• directional — field-level trends/consensus: "The field shifted from monolithic context to progressive disclosure in 2025"

Target mix: ~40% empirical, ~30% architectural, ~15% directional, ~15% comparative.

Confidence Levels

• verified — evidence visible in source (benchmark code, architecture diagram, specific file reference)
• reported — cited in source but not independently validated (README claims, self-reported benchmarks). This is the default.
• inferred — synthesized across multiple sources, not stated directly in any single one. Target 10-20% of claims.

Source Attribution

Extract actual raw/ paths from [Source](../raw/...) links near each claim.

CRITICAL: match each claim to the source that ACTUALLY contains its evidence, not just the nearest citation in the article. If a paragraph cites source A but the specific number comes from source B cited two paragraphs earlier, attribute to source B.

Leave source_refs empty rather than guessing. Wrong attribution is worse than missing attribution.

Prefer deep/ sources for implementation claims. If a claim describes architecture, algorithms, data structures, or specific benchmark methodology, the evidence is almost certainly in raw/deep/repos/ or raw/deep/papers/, not the shallow README. Check both — cite the one that contains the detail.

If build/lessons.md exists, read it for known attribution patterns from previous compilations.

Temporal Scope

For time-sensitive data (star counts, benchmark results, adoption numbers), set temporal_scope to "as of YYYY-MM". For timeless architectural facts, set to null.

Output

Write build/claims.json. See references/schema.md for the exact schema.

Phase 2: Self-Eval

Sample 30 claims that have source refs, stratified proportionally by claim type (e.g., if 40% of claims are empirical, sample ~12 empirical).

For each sampled claim:

1. Read the cited raw source file
2. Check whether the source EXPLICITLY supports the claim
3. Verdict: PASS if the source contains specific evidence. FAIL if not.
4. Semantic equivalence counts: "four-level" = "Levels 0-3", "51.8K stars" = "51,880 stars". Only FAIL when evidence is genuinely missing or contradicted.

Output

Write build/eval-report.json with:

• Overall accuracy rate
• Breakdowns by claim type and by bucket
• Details for every sampled claim (verdict + reason)
• Full details for every failure

See references/schema.md for the exact schema.

Changelog Entry

Append a summary line to wiki/CHANGELOG.md:

## YYYY-MM-DD
- Claims extraction: {total} claims ({empirical}E/{architectural}A/{comparative}C/{directional}D)
- Self-eval: {accuracy}% accuracy on {sample_size} sampled claims
- Failures: {count} ({brief list of failure reasons})

Quality Rules

• Never fabricate a source path. If you cannot find the source for a claim, leave source_refs empty.
• Every claim must be independently verifiable by reading the cited source.
• Prefer specific numbers over vague statements. "26% improvement" not "significant improvement".
• Flag conflicting claims between articles rather than silently picking one.