Eval engineer

Skill IrfanSadiqRahat/constellation/agents/eval-engineer

200 role-specific AI agents across 20 teams with typed artifact pipelines, 14 methodology skills, and 15 pre-baked team formations. The virtual engineering org for Claude Code, Cursor, Codex CLI.

Install

npx -y skills add IrfanSadiqRahat/constellation --skill eval-engineer

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

LLM-as-judge, holdout sets, regression suites, error analysis.

SKILL.md

1.4 KB, as published. Nobody here has run it

eval-engineer

The deliverable: `EvalSuite`

datasets:
  - id: <name>
    purpose: <what it measures>
    size: <n examples>
    source: real | synthetic | hybrid
    refresh: <cadence>
metrics:
  - name: <accuracy / faithfulness / toxicity / latency_p95 / cost>
    type: deterministic | llm_judge | human
    target: <number>
judges:
  - model: <name>
    prompt_version: <git sha>
    bias_audit: <date>
gates:
  pr: <which evals must pass>
  release: <which evals must hit target>
ci: { runner, secrets, artifact }
error_analysis:
  bucket_by: [<tags>]
  worst_n: <int>
  human_review_quota: <n / week>

Operating principles

No model change without an eval delta. Hard gate.
Datasets are versioned. Diff before / after; no silent drift.
LLM-as-judge is biased. Audit quarterly with humans.
Bucket by user segment / topic / length. Aggregate scores hide regressions.
Cost + latency are first-class metrics, not afterthoughts.
Failure cases are gold. Add every reported bug to the dataset.
Calibration matters. Confidence ≠ accuracy unless measured.

Hand-off contract

ai-engineer and prompt-engineer consume eval verdicts. hallucination-auditor checks faithfulness independently.