Judge

Skill tmuskal/arc-agi-benchmarker/plugins/longmemeval-benchmarker/skills/judge

Prove you achieved AGI at home by testing your claude-code setup against arc-agi-3 benchmarks

Install

npx -y skills add tmuskal/arc-agi-benchmarker --skill judge

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

LLM-as-judge shim for LongMemEval - wraps upstream get_anscheck_prompt and calls Anthropic (default) or OpenAI (fallback) with exponential backoff

SKILL.md

2.2 KB, as published. Nobody here has run it

LongMemEval Judge

You are invoking the judge shim manually (e.g. to re-score an existing item-results.jsonl with a different judge model). Normally the run-benchmark skill calls the judge inline.

Step 1: Resolve venv and config (same pattern as other skills).

Step 2: Choose provider + model

Default: anthropic / claude-opus-4-6.
Fallback: openai / gpt-4o.
The shim auto-falls-back if the chosen provider's API key is unset, or if the call fails after 6 backoff tries.

Step 3: Single-item smoke test

$VENV_PYTHON -c "
import sys, json
sys.path.insert(0, 'plugins/longmemeval-benchmarker/scripts')
from judge_shim import judge
from pathlib import Path

out = judge(
    question_type='single-session-user',
    question='What is my dog\\'s name?',
    answer='Rex',
    hypothesis='Your dog is named Rex.',
    longmemeval_root=Path('longmemeval'),
    provider='anthropic',
    model='claude-opus-4-6',
)
print(json.dumps(out, indent=2))
"

Step 4: Re-judge an existing run

Iterate over item-results.jsonl, call judge() with a new model, write to item-results.<newmodel>.jsonl alongside the original (do NOT overwrite).

$VENV_PYTHON -c "
import sys, json
from pathlib import Path
sys.path.insert(0, 'plugins/longmemeval-benchmarker/scripts')
from judge_shim import judge
from checkpoint_io import append_jsonl

run_dir = Path('.longmemeval-benchmarks/runs/<RUN_ID>')
out_path = run_dir / 'item-results.rejudge.jsonl'
for line in (run_dir / 'item-results.jsonl').read_text().splitlines():
    r = json.loads(line)
    j = judge(r['question_type'], r['question'], r['answer'], r['hypothesis'],
              Path('longmemeval'), provider='openai', model='gpt-4o')
    r['judgment'] = {'model': j['model'], 'label': j['label'], 'raw': j['raw']}
    append_jsonl(out_path, r)
print('wrote', out_path)
"

Notes

The shim imports get_anscheck_prompt from longmemeval/src/evaluation/evaluate_qa.py (L21-47). Do not reimplement.
backoff.expo with max_tries=6 + full jitter is used on every API call.