Repo bullshit meter

Repo Bullshit Meter

Assess claims against evidence. Every claim gets a source, an implementation check, and a proof trail; never judge popularity, intent, or aesthetics — only whether the claim holds.

Non-Negotiables

• Keep guess count at 0. When evidence is missing, say unknown, not proven, or blocked — never infer.
• Claims come from the README, docs, website, marketplace listings, release notes, package metadata, demos, benchmarks, and issues.
• Every verdict needs a claim source, repo evidence, and feasibility evidence — not inference.
• Do NOT execute untrusted repo code: no scripts, agents, MCP servers, or networked components. Static inspection only. Credentials/secrets are out of scope.
• Popularity is not proof: stars, forks, slick docs, mascots, and badges count for nothing.
• Blunt labels describe evidence quality only, not the author.

Workflow

0. Select audit profile

   • Default to standard. Do not prompt for depth; state the chosen profile in the readout and move on.
   • Use quick only when the user implies speed ("quick check", "gut check", "at a glance"). Use deep only when they ask for exhaustive validation, ecosystem/package checks, CI history, or external compatibility proof.
   • quick: claim audit via README + source grep, key files, no external research.
   • standard: quick plus tests/CI inspection, docs cross-check, safe syntax/static checks, official platform/protocol docs.
   • deep: standard plus package registries, releases/issues/CI logs, external docs; still static-only.

1. Lock target

   • Normalize the target as OWNER/REPO, URL, package, or local checkout.
   • Check the verdict ledger for a prior audit before doing new work (run from this skill's directory). Pass the repo's current pushedAt from gh repo view and the tool decides:

     python scripts/render_meter.py --history OWNER/REPO --pushed-at <current pushedAt>
     

   • CACHE HIT means the repo is unchanged since the last audit: present the cached verdict and ask whether a re-audit is wanted. CHANGED means re-audit and report score drift against the prior entry.
   • If the repo is private, missing, renamed, archived, or ambiguous, say so and stop.

2. Capture claims

   • Inspect the README, docs, homepage, marketplace/listing pages, release notes, and package metadata.
   • Quote claims exactly; prefer URL anchors or file:line sources.
   • Rank claims by materiality: capability, integration/compatibility, performance/cost, security/privacy, setup/usability, demo/benchmark.
   • Cap the scorecard at the ~8 most material claims. Fold trivial or duplicate claims into one line under Blockers ("N minor claims not individually audited") instead of scoring them — a 10-claim scorecard with severities 5→2 beats a 25-claim one.
   • For files over ~10k tokens, inspect targeted sections instead of full reads.

   Token discipline (applies to every step): read only the files a scored claim depends on — never crawl the whole repo. Prefer rg for locating symbols over reading files; read matched regions, not whole files. Never re-read a file already inspected. One evidence-gathering pass per claim; if evidence isn't found after two targeted searches, score unverifiable and move on rather than widening the search.

3. Gather repo evidence

   • Live metadata via GitHub CLI:

     gh repo view OWNER/REPO --json nameWithOwner,description,homepageUrl,defaultBranchRef,createdAt,updatedAt,pushedAt,isArchived,isFork,licenseInfo,latestRelease,repositoryTopics,stargazerCount,forkCount
     gh api repos/OWNER/REPO/releases?per_page=10
     gh api repos/OWNER/REPO/commits?per_page=10
     

   • Read-only shallow clone into a temp directory:

     git clone --depth 1 https://github.com/OWNER/REPO.git <tempdir>/repo-bullshit-meter/OWNER-REPO
     

   • Search with rg (ripgrep): manifests, entrypoints, tests, docs, and the specific symbols each claim depends on.

4. Gather external feasibility evidence (standard and deep)

   • Verify platform/protocol claims against official docs using your environment's web search or documentation tooling (web search, an MCP docs server such as Context7, or plain fetch).
   • For deep, also check package registries (npm/PyPI/crates), release history, and issue trackers for contradicting reports.

5. Apply claim-specific checks

   • MCP/agent-extension claims: valid manifest/server entrypoint, transport, client compatibility, setup path.
   • Token-saving claims: identify the actual mechanism (smaller prompts, retrieval, summarization, pruning). "Saves N% tokens" without a measurement mechanism scores unsupported.
   • Security/privacy claims: code/config proof plus threat-model fit. Marketing language is not proof.
   • Performance/accuracy claims: reproducible benchmark, dataset, and method — or unverifiable.

6. Score

   • Use references/rubric.md: verdict weight × severity, weighted average.
   • Penalize contradictions harder than missing polish. Insufficient evidence is not fraud.

7. Render the readout

   • Write the evidence JSON (schema below), then render with source verification against the clone:

     python scripts/render_meter.py evidence.json --format report --ledger --verify-sources <clone_root>
     

   • The renderer refuses any claim without a source and a non-empty evidence list. With --verify-sources, it also checks every file:line claim source against the actual checkout (file exists, line in range) and refuses to render fabricated citations. Always pass it when you have a clone.
   • --ledger appends the verdict (repo, date, score, label, confidence, claim counts, evidence path) to ledger.jsonl in the skill root so future audits can detect drift. Pass --pushed-at <ISO> with the repo's pushedAt when known. Use --ledger on every completed audit; omit for dry runs.
   • Use --format compact for short chat replies and --format table for a dense spreadsheet-like claim table.
   • Pass --ascii (or set RBM_ASCII=1) if the terminal lacks a Nerd Font.

Evidence JSON schema

{
  "repo": "OWNER/REPO",
  "commit": "branch@sha",
  "score": 62,
  "confidence": "medium",
  "audit_profile": "standard",
  "summary": "One sentence verdict.",
  "headline": "Short human-readable title.",
  "bottom_line": "What a user should actually do about the core claim.",
  "verification": {
    "passed": ["python -m compileall -q src"],
    "blocked": ["No live API key available"],
    "not_run": ["runtime install", "live API calls"],
    "commands": [], "urls": [], "files": []
  },
  "claims": [
    {
      "claim": "Saves 70% of tokens per session.",
      "claim_source": "README.md:12",
      "verdict": "unsupported",
      "severity": 5,
      "evidence": [
        "src/index.ts implements caching only; no token accounting found",
        "no benchmark or dataset in repo (searched: bench, eval, tokens)"
      ],
      "note": "Mechanism described is caching, not token reduction."
    }
  ]
}

Verdicts: proven, plausible, partial, unverifiable, unsupported, contradicted, impossible.

Output contract

Every readout includes:

• The rendered meter (boxed banner with score, label, confidence, claim bars).
• A per-claim scorecard: claim, source, evidence, verdict, note.
• The verification ledger: what ran, what was blocked, what was not run, URLs/files inspected.
• Blockers and uncertainty stated plainly.
• No claim of runtime/browser/API verification that did not happen.
• In the verification ledger, describe external research by the sources checked (URLs, docs, registries) — never by the names of local helper tools, wrappers, or scripts that are not part of this skill. The readout must make sense to someone who has only this repo.