Karvey benchmark models

Skill MauricioQuezadaHaintech/karvey/plugins/karvey/skills/karvey-benchmark-models

Karvey — método spec-driven development agnóstico de stack (Afán, selknam). Plugin de Claude Code. © HainTech, Apache 2.0.

Install

npx -y skills add MauricioQuezadaHaintech/karvey --skill karvey-benchmark-models

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Cross-model benchmark for the Karvey method. Side-by-side comparison of models (e.g. Claude vs GPT vs Gemini) on a skill or task — latency, tokens, cost, and optional LLM-judged quality. Triggers include "karvey benchmark models", "comparar modelos", "compare models", "benchmark de modelos", "model benchmark", "qué modelo conviene", "which model to use", "latencia tokens costo", "latency tokens cost".

SKILL.md

3.3 KB, as published. Nobody here has run it

Karvey — Model Benchmark (cross-model)

CROSS-CUTTING SKILL of the Karvey Method. It is a support layer, NOT a phase. It does not advance or modify the method's cycle: it never touches spec.json:phase. It can be invoked at any time without altering the project's state.

Inspired by gstack /benchmark-models: a side-by-side comparison of models (for example Claude vs GPT vs Gemini) over the same skill or task, measuring latency, tokens, cost, and, optionally, quality judged by an LLM.

Purpose

Decide with data which model is best for a given task or skill in the project. Instead of choosing a model by intuition or habit, this skill runs the same task across several models and delivers an objective comparison table plus a well-founded recommendation.

It is a meta / diagnostic skill: it observes and measures, but it does not modify the project's code or produce artifacts from the method's phases. It generates no commits, does not change the solution's files, does not advance the phase.

Steps

Define the task and the models to compare.
- Identify the target task or skill (from the argument or by asking the user).
- Detect the model CLIs available in the environment: claude, codex/gpt, gemini, etc.
- Degrade gracefully: if a CLI is not accessible, exclude it from the benchmark and warn. Compare only the models actually available.
Run the same task on each model.
- Use exactly the same prompt/input for all, so the comparison is fair.
- Run in isolation per model and capture the full output of each one.
Measure latency, tokens, and cost.
- Latency: wall-clock time of each run.
- Tokens: input + output reported by each CLI.
- Cost: estimate from tokens and each model's current price.
(Optional) Quality judged by an LLM-judge.
- If quality needs to be evaluated, use a model as judge to score the outputs against an agreed rubric (accuracy, completeness, format, etc.).
- Keep the judge fixed and the rubric explicit so the scores are comparable.
Comparison table + recommendation.
- Present a side-by-side table: model, latency, tokens, cost, and (if applicable) quality.
- Close with a clear recommendation of which model is best for that task and why (cost/quality/latency balance per the objective).

Reminders

Does not advance the phase. This skill never writes spec.json:phase or triggers method transitions.
Does not touch the project's code. It only reads, runs test runs, and reports.
It is invocable at any point in the Karvey cycle as support for decision-making.

Part of the Karvey™ Method — © HainTech, by Mauricio Quezada Ibáñez · Apache 2.0 · see karvey/LICENSE and karvey/TRADEMARK.md.