Puzzletide agent evals

Skill Caravaca-Labs/puzzletide-cli/skills/puzzletide-agent-evals

Word search generator, crossword generator, and sudoku generator & solver in one local-first CLI — printable PDF puzzle worksheets, themed word banks, verifiable LLM evals, and agent skills. From the makers of puzzletide.com.

Install

npx -y skills add Caravaca-Labs/puzzletide-cli --skill puzzletide-agent-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

20 days oldThe repository was created 20 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use this skill when the user wants verifiable reasoning tasks to benchmark or test an LLM or agent — reproducible puzzle task sets (sudoku, word search) with objective, by-construction grading. No answer key to trust: answers are verified against the rules and the grid.

SKILL.md

3.0 KB, as published. Nobody here has run it

PuzzleTide Agent Evals

Generate reproducible, objectively gradable puzzle tasks for testing models and agents with the local PuzzleTide CLI.

Why puzzles: they are verifiable by construction. A sudoku answer either satisfies the rules and preserves the givens or it doesn't; a word search answer either spells the word along a straight line in the grid or it doesn't. Grading needs no LLM judge and no trusted answer key.

Prefer the local CLI. Check availability in this order:

ptide --version
puzzletide --version
npx puzzletide --version

If none of those work, ask the user before installing (npm install -g puzzletide).

Generate a task set

ptide eval generate --type sudoku --n 20 --difficulty hard --seed 1 --out tasks.json
ptide eval generate --type wordsearch --n 10 --difficulty medium --seed 1 --out tasks.json

The tuple (type, difficulty, n, seed) fully determines the task set, so it names a reproducible benchmark — same command, same tasks, on any machine.

Each task has id, instructions, and the puzzle payload:

sudoku: puzzle (81 chars, . = empty). Expected answer: completed 81-char string.
wordsearch: grid (array of row strings) and words. Expected answer: JSON array of {word, startRow, startCol, endRow, endCol} (0-indexed).

Run the subject model

Send each task's instructions + payload to the model under test and collect answers as a JSON array of {id, answer}.

Grade

ptide eval check --tasks tasks.json --answers answers.json --json

Returns per-task pass/fail with reasons and a summary score. Grading is deterministic and local.

Safety

Everything runs locally; no account, API key, or network access.
Do not install packages without asking the user first.