Report

Skill tmuskal/arc-agi-benchmarker/plugins/longmemeval-benchmarker/skills/report

Prove you achieved AGI at home by testing your claude-code setup against arc-agi-3 benchmarks

Install

npx -y skills add tmuskal/arc-agi-benchmarker --skill report

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Generate a scorecard from a completed LongMemEval run - computes overall accuracy + per-question-type accuracy and writes scorecard.json

SKILL.md

2.0 KB, as published. Nobody here has run it

LongMemEval Report

Step 1: Resolve venv and config.

Step 2: Parse args

Argument	Default	Notes
`<run_id>`	`latest`	UUID or `latest`
`--format`	`markdown`	`markdown` / `json` / `summary`

Step 3: Resolve `latest`

Pick the most recently modified directory under .longmemeval-benchmarks/runs/.

$VENV_PYTHON -c "
from pathlib import Path
d = Path('.longmemeval-benchmarks/runs')
runs = sorted([p for p in d.iterdir() if p.is_dir()], key=lambda p: p.stat().st_mtime, reverse=True)
print(runs[0].name if runs else '')
"

Step 4: Compute + write scorecard.json

$VENV_PYTHON -c "
import json, sys
from pathlib import Path
sys.path.insert(0, 'plugins/longmemeval-benchmarker/scripts')
from scorecard import compute
from checkpoint_io import write_atomic_json

run_id = '<RUN_ID>'
run_dir = Path('.longmemeval-benchmarks/runs') / run_id
meta = json.load(open(run_dir / 'run-meta.json'))
variant = meta.get('datasetVariant', 'longmemeval_s')
cfg = json.load(open('.longmemeval-benchmarks/config.json'))
n_total = len(json.load(open(cfg['datasetPath'])))
card = compute(run_dir, n_total, variant, run_id)
write_atomic_json(run_dir / 'scorecard.json', card)
print(json.dumps(card, indent=2))
"

Step 5: Render markdown

# LongMemEval Run {runId}

- Dataset: {datasetVariant}
- Evaluated: {n_evaluated} / {n_total}
- Overall accuracy: {overall_accuracy:.3f}

## Per question-type accuracy

| Type | Accuracy | n |
|---|---|---|
| single-session-user | ... | ... |
| single-session-assistant | ... | ... |
| multi-session | ... | ... |
| temporal-reasoning | ... | ... |
| knowledge-update | ... | ... |
| preference | ... | ... |
| abstention | ... | ... |

For --format json, print scorecard.json raw. For --format summary, print a single line: runId variant n_eval/n_total overall_acc.