Puzzletide agent evals

Skill Caravaca-Labs/puzzletide-cli/skills/puzzletide-agent-evals

Use this skill when the user wants verifiable reasoning tasks to benchmark or test an LLM or agent — reproducible puzzle task sets (sudoku, word search) with objective, by-construction grading. No answer key to trust: answers are verified against the rules and the grid.From its SKILL.md

Install

npx -y skills add Caravaca-Labs/puzzletide-cli --skill puzzletide-agent-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.0 KB, 579 tokens by cl100k_base, as published. Nobody here has run it

PuzzleTide Agent Evals

Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.

Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.

Prefer the local CLI. Check availability in this order:

ptide --version
puzzletide --version
npx puzzletide --version

If none of those work, ask the user before installing (npm install -g puzzletide).

Generate a task set

ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json

The tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.

Each task has id, instructions, and the puzzle payload:

sudoku: puzzle (81 chars, . = empty). Expected answer: completed 81-char string.
wordsearch: grid (array of row strings) and words. Expected answer: JSON array of {word, startRow, startCol, endRow, endCol} (0-indexed).

Run the subject model

Send each task's instructions + payload to the model under test and collect answers as a JSON array of {id, answer}.

Grade

ptide eval check --tasks tasks.json --answers answers.json --json

Returns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.

Safety

Everything runs locally; no account, API key, or network access.
Do not install packages without asking the user first.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.