Graph eval

Skill ayaangazali/graph-engineering/skills/graph-eval

Offline, code-gated A/B harness that ranks candidate finder briefs by how many synthetic planted defects each catches — a relative prompt-selection signal, never a recall estimate. Off the critical path.From its SKILL.md

Install

npx -y skills add ayaangazali/graph-engineering --skill graph-eval

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.0 KB, as published. Nobody here has run it

Graph-eval: pick the better brief with a code-measured A/B, honestly

Rank candidate finder briefs (or DAG shapes) for a task class by relative performance on style-matched synthetic plants. This is a disciplined prompt-A/B harness — not a categorical capability and not a measurement of how well /graph will do on the user's real repo. Run it offline; never on the critical path of a real goal.

Task class: $ARGUMENTS

What this is (and the honest limits, which are load-bearing)

Output is: "brief A caught more synthetic plants than brief B — a relative selection signal." It is never "brief A has X% recall on your repo." Correlated same-model finders with heterogeneous catchability cannot estimate their own real-world recall; do not compute Lincoln-Petersen / capture-recapture bands — they launder a wide-variance guess as an instrument reading.
Because plants are synthetic, this is Goodhart-prone. The guardrails below are the point of the skill, not decoration.

Procedure

Mutually-blind plant / hunt split. A plant agent (spawn with isolation: 'worktree') injects K defects into a copy of the target and returns an answer key that only orchestrating code ever sees — the key is never concatenated into any finder/hunter prompt. Require K ≥ 30 so a win clears noise.
Held-out archetype split (anti-Goodhart). Keep an escape_catalog of defect archetypes. The archetypes used to test a brief are never the archetypes named in that brief. A brief that only finds what it was told to look for is not better; it's overfit.
Run each candidate brief as a fan-out over the planted copy (same seed set, same budget). Each hunter returns found locations; code scores them against the held answer key. No agent grades itself.
Promotion is code-gated. Compute the delta (plants caught A − B) in code. Promote a champion only if the delta is positive across the K ≥ 30 plants and the held-out archetypes — never on an agent's self-report. Ties or noise-level deltas promote nothing.
Persist the champion to .claude/graph-memory/<task-class>.md (a git-committable brief), with a one-line record of the measured delta and the plant count. The /graph skill may read this file as a prior for that task class — clearly labeled as a synthetic-plant selection signal, not a guarantee.

Output

A short report: candidate briefs ranked by code-measured plants-caught, the delta, K, the held-out archetypes used, and the promoted champion path (or "no promotion — delta within noise"). State plainly: "relative prompt-selection on synthetic plants; not a recall figure for any real repo."