Compare runs

Skill tmuskal/arc-agi-benchmarker/plugins/longmemeval-benchmarker/skills/compare-runs

Prove you achieved AGI at home by testing your claude-code setup against arc-agi-3 benchmarks

Install

npx -y skills add tmuskal/arc-agi-benchmarker --skill compare-runs

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Compare two LongMemEval runs - diff scorecards, harness_config, and per-question-type accuracy

SKILL.md

1.4 KB, as published. Nobody here has run it

LongMemEval Compare Runs

Step 1: Resolve venv. Parse two positional args: `<runIdA> <runIdB>`.

Step 2: Load both run directories

$VENV_PYTHON -c "
import json
from pathlib import Path

def load(run_id):
    d = Path('.longmemeval-benchmarks/runs') / run_id
    return {
        'meta': json.load(open(d / 'run-meta.json')),
        'card': json.load(open(d / 'scorecard.json')),
    }

A = load('<RUN_A>')
B = load('<RUN_B>')
print(json.dumps({'A': A, 'B': B}, indent=2))
"

If scorecard.json is missing for a run, invoke the report skill on it first.

Step 3: Produce diff table

Print markdown:

# Run Comparison

|  | A ({runA}) | B ({runB}) | Delta |
|---|---|---|---|
| variant | ... | ... | |
| target model | ... | ... | |
| judge model | ... | ... | |
| n_evaluated | ... | ... | +/- |
| overall_accuracy | ... | ... | +/-.XXX |
| single-session-user | ... | ... | +/- |
| single-session-assistant | ... | ... | +/- |
| multi-session | ... | ... | +/- |
| temporal-reasoning | ... | ... | +/- |
| knowledge-update | ... | ... | +/- |
| preference | ... | ... | +/- |
| abstention | ... | ... | +/- |

Also diff harness_config (model / plugins / skills / mcp_servers) and surface added/removed entries.

Step 4: Highlights

Print the top-3 question types with the largest positive and negative deltas.