agentsclimarketplace

Doc accuracy

Skill rjmurillo/ai-agents/.claude/skills/doc-accuracy

Multi-phase documentation verification treating code as source of truth. Use when you say "check documentation accuracy", "verify code examples compile", "audit docs vs code", "check doc consistency", or "run doc-accuracy". Use for pre-release doc audits and checking behavioral claims. Do NOT use for broad codebase analysis (use analyze) or fixing markdown fence syntax (use fix-markdown-fences).From its SKILL.md

Install
npx -y skills add rjmurillo/ai-agents --skill doc-accuracy

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

What its file declares

Copied from the file, not written here

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

9.1 KB, as published. Nobody here has run it

Documentation Accuracy Skill

Verify documentation claims against actual code behavior. Code is truth; docs are the subject under test.

Triggers

Trigger PhraseOperation
check documentation accuracyFull audit (Phases 1-6)
verify code examplesCompilability check (Phases 1-3)
audit docs vs codeBehavioral verification (Phases 1-4)
check doc consistencyCross-document consistency (Phases 1-2, 5)
run doc-accuracyFull audit (Phases 1-6)

When to Use

Use this skill when:

  • Documentation may contain non-compilable code examples
  • Behavioral claims in docs may contradict implementation
  • Quantitative claims (performance, limits) appear in multiple files with different values
  • Preparing for a release and need a documentation accuracy audit
  • API reference may be missing public members

Use direct code review instead when:

  • Investigating a single known documentation error
  • The inaccuracy is already identified and needs a fix

Replaces

SkillReason
incoherence15.8% recall on critical issues; Haiku agents too shallow
doc-coverage0% recall on actionable issues; checks presence, not correctness
doc-syncNo scripts, purely manual LLM workflow
comment-analyzerAdvisory only, single-file scope

Architecture

Asymmetric Verification

Code compiles and runs. Documentation describes what code does. When they disagree, the code is right. This skill reads code first, builds a verified model, then checks documentation claims against that model.

Phase Overview

Phase 1: Assessment        (script-only, <30s)  -> assessment.json
Phase 2: Claim Extraction  (script-only, <15s)  -> claims.json
Phase 3: Compilability     (script-only, <60s)  -> compilability-findings.json
Phase 4: Behavioral        (Sonnet agents, 3-7m) -> behavioral-findings.json
Phase 5: Cross-Document    (script + Sonnet, 1-2m) -> consistency-findings.json
Phase 6: Structure         (Sonnet agent, 1-2m)  -> structure-findings.json

Scripts

ScriptPurpose
scripts/doc_accuracy.pyPhases 1-3: Assessment, claim extraction, compilability check

Invocation

# Full deterministic scan (Phases 1-3)
python3 scripts/doc_accuracy.py --target /path/to/repo

# Compilability only
python3 scripts/doc_accuracy.py --target /path/to/repo --phases 3

# Incremental (changed files only)
python3 scripts/doc_accuracy.py --target /path/to/repo --diff-base main

# JSON output to specific directory
python3 scripts/doc_accuracy.py --target /path/to/repo --output-dir .doc-accuracy

# Set severity threshold for exit code
python3 scripts/doc_accuracy.py --target /path/to/repo --severity-threshold critical

# Markdown report output
python3 scripts/doc_accuracy.py --target /path/to/repo --format markdown

# Text summary to stdout
python3 scripts/doc_accuracy.py --target /path/to/repo --format summary

Output Artifacts

FileDescription
assessment.jsonPhase 1: doc/source inventory with symbol index
claims.jsonPhase 2: verifiable claims extracted from docs
compilability-findings.jsonPhase 3: run status and symbol resolution findings
gate-result.jsonGate verdict, reason when inconclusive, and severity counts
report.mdMarkdown summary (when --format markdown)

Process

Phases 1-3: Deterministic (Script)

Run doc_accuracy.py to produce JSON artifacts. No LLM calls.

  1. Phase 1 (Assessment): Enumerate documentation and source files. Extract public symbols via regex. Build doc-to-source mapping.
  2. Phase 2 (Claim Extraction): Parse markdown files. Extract verifiable claims with file path, line number, claim type, and referenced symbols.
  3. Phase 3 (Compilability): Verify type names, method names, parameter names in code examples exist in the codebase via symbol index lookup. If Phase 1 finds no source symbols, report DID_NOT_RUN with zero findings instead of treating every documentation reference as unresolved.

Phase 4: Behavioral Verification (Agent)

Dispatch one Sonnet agent per file group. Each agent receives:

  • Full source file content (read code FIRST)
  • Documentation file content
  • Filtered claims from claims.json
  • Compilability findings (to avoid re-checking)

Agent prompt:

You are verifying documentation accuracy. Code is the source of truth.

SOURCE FILES (read these first):
[full content of mapped source files]

DOCUMENTATION FILE:
[full content of the documentation file]

CLAIMS TO VERIFY:
[filtered claims from claims.json]

For each claim:
1. Find the relevant code in the source files
2. Determine if the documentation claim accurately describes the code behavior
3. If inaccurate: severity, description, evidence (with line numbers), suggested fix
4. If accurate: mark as PASS

Additionally check for:
- Public API members in source absent from documentation
- Behavioral nuances the documentation omits or misrepresents
- Default values that differ between docs and code

Launch agents in parallel (one per file group).

Phase 5: Cross-Document Consistency (Agent)

From claims.json, filter to quantitative and behavioral claims. Group by topic. For groups with conflicting values across files, dispatch a Sonnet agent with benchmark data to determine the correct value.

Phase 6: Structure and Quality (Agent)

Validate documentation structure (indexes, navigation, completeness). Apply comment quality framework (accuracy, completeness, long-term value, misleading elements, improvements) to a 20% sample of source comments.

Reconciliation (Interactive)

Present proposed fixes for user approval before modifying any file. Categories:

  • Documentation prose fixes
  • Code example corrections
  • Consistency resolution (single source of truth)
  • Structural updates (indexes, navigation)

Issue Taxonomy

ClassDescriptionDetection Phase
1: Spec vs BehaviorDocs say X, code does YPhase 4
2: Non-Compilable CodeCode examples reference nonexistent symbolsPhase 3
3: Cross-Doc InconsistencySame fact, different values across filesPhase 5
4: Domain ViolationsTechnology convention violations (OTel, Prometheus)Phase 4 + Plugins
5: API Surface GapsPublic API exists but is undocumentedPhase 3 + Phase 4

Severity Levels

LevelDefinition
CriticalCode will not compile, or behavior is silently wrong
HighMaterially misleading but no immediate failure
MediumInconsistent or confusing but correct in at least one location
LowCosmetic, improvement opportunity, or minor omission

Exit Codes

Canonical source bundled with this skill:

scripts/doc_accuracy.py
Exit Codes:
    0: No findings at or above severity threshold
    1: Error or inconclusive run, including no source symbols for Phase 3
    2: Configuration error, including an invalid --diff-base
    3: External dependency failure, including unavailable or failed Git
    10: Findings at or above severity threshold

Anti-Patterns

AvoidWhyInstead
Trusting documentation as peer of codeDocs and code are not equal; code is truthAlways read implementation before checking docs
Using Haiku for behavioral verification15.8% recall vs 100% with SonnetUse Sonnet agents for Phase 4
One agent per dimensionLoses cross-cutting contextOne agent per file group
Skipping Phase 1 for Phase 4Agents need symbol index for precise verificationAlways run Phases 1-2 first
Running all phases on unchanged filesWastes time and tokensUse --diff-base for incremental checks

Verification

After running:

  • All JSON artifacts created in output directory
  • Every finding has file path, line number, severity, and evidence
  • Code examples verified against actual method signatures
  • Behavioral claims verified by reading implementation source
  • Cross-document conflicts identified with all locations listed
  • Exit code reflects severity threshold
  • Phase 3 reports DID_NOT_RUN when no source symbols exist

Related Skills

SkillRelationship
incoherenceReplaced: Detection logic superseded by Phases 3-5
doc-coverageReplaced: Symbol extraction logic preserved in Phase 1
doc-syncReplaced: Structural audit absorbed into Phase 6
analyzeComplementary: broader codebase analysis
style-enforcementComplementary: code style checks

Keep looking

Skills are one crate of 615,528. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.