Eval
Run gaslighter's eval suite to measure requirement completion across baseline and nudge arms.From its SKILL.md
npx -y skills add LarryGF/gaslighter --skill evalAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 5 stars5 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
1.5 KB, as published. Nobody here has run it
Gaslighter Eval
Resolve <plugin-root> from this loaded skill's location. Run every command below from <plugin-root>; do not assume the caller's cwd is the gaslighter checkout.
cd "<plugin-root>/evals" && python3 run.py <arguments>
Execution
-
Parse user arguments; use
config.jsondefaults when absent. -
For
--selftest, run in the foreground. -
Otherwise launch the command through the host's background-task or supervised-process facility; report its task identifier and configuration.
-
On completion, run:
cd "<plugin-root>/evals" && python3 analyze.py summary runs/<timestamp>
5. Then load the `judge` skill for `runs/<timestamp>` automatically. Do not tell the user to do that manually.
Use the host's shell, task, and skill mechanisms; do not assume Claude-specific tool names.
## Arms
- `baseline` — no plugin.
- `gaslighter-off` — plugin loaded but disabled.
- `gaslighter-lite` — soft nudge.
- `gaslighter-full` — hard block.
- `nudge-prompt` — static prompt-only control.
## Flags
- `--selftest` — validate scorers without API spend.
- `--all` — run all tasks.
- `--task TASK` / `--exclude-task TASK` — select tasks.
- `--models MODELS` — `haiku`, `sonnet`, `opus`.
- `--runs N`, `--workers N`, `--timeout N`.
- `--config PATH`.
- `--plugin-dir PATH`.
Results are written under `evals/runs/<timestamp>/`: `results.json`, `summary.json`, and preserved workspaces.