Eval
Quality and performance evaluation with baseline comparison. Sub-modes: correctness, performance, quality, regression. Outputs PASS/WARN/FAIL per dimension. Use for pre-ship evaluation or regression checks.From its SKILL.md
npx -y skills add epicsagas/epic-harness --skill evalAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 22 stars22 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
7.9 KB, as published. Nobody here has run it
Eval — Quality & Regression Gate
CRITICAL: Run HARNESS_DIR=$(epic path) first. Never use .harness/ in the project directory.
When to Trigger
- Before
/shipcreates a PR (automatic if eval.yaml exists) - After
/gocompletes a feature - On explicit
/evalcommand - When user mentions "regression", "baseline", "eval suite", "quality check"
- CI:
make evalorepic eval --json
Execution Modes
4 dimensions run in parallel where possible:
- eval:correctness — Test pass rate, mutation score, assertion density
- eval:performance — Throughput, latency, memory (opt-in)
- eval:quality — Lint, code quality, LLM-as-judge
- eval:regression — Baseline comparison, score deltas
Process
Step 0: Prerequisites
HARNESS_DIR=$(epic path)
If $HARNESS_DIR/eval/eval.yaml does not exist, run scaffold:
epic eval --init
Read the config:
cat $HARNESS_DIR/eval/eval.yaml
Step 0.5: Scaffold benchmarks (when no benchmark infrastructure exists)
If eval.yaml has benchmarks: [] and no benchmark files are found in the project:
-
Generate stub files using the CLI:
epic eval --scaffoldSupported stacks (auto-detected from project markers):
Stack Detected by Generated file Output format Rust Cargo.tomlbenches/eval_harness.rscriterion (exit code) Python pyproject.toml/setup.pybenchmarks/eval_runner.pyJSON composite TypeScript tsconfig.jsonbenchmarks/eval.tsJSON composite Node.js package.jsonbenchmarks/eval.mjsJSON composite Go go.modbenchmarks/eval_test.goJSON composite Java pom.xml/build.gradlebenchmarks/EvalBenchmark.javaexit code Kotlin build.gradle.ktsbenchmarks/EvalBenchmark.ktexit code Ruby Gemfilebenchmarks/eval_benchmark.rbJSON composite PHP composer.jsonbenchmarks/eval_benchmark.phpJSON composite C# *.csproj/*.slnBenchmarks/EvalBenchmark.csJSON composite Swift Package.swiftbenchmarks/EvalBenchmark.swiftJSON composite Elixir mix.exsbenchmarks/eval_benchmark.exsJSON composite C++ CMakeLists.txtbenchmarks/eval_benchmark.cppexit code -
Customize the generated file — every file has
# TODO/// TODOmarkers:- Replace placeholder logic with calls to your actual domain functions
- Adjust the composite score weights to reflect your domain priorities
- For precision/recall benchmarks: wire in your real test set and model
-
If
--scaffoldcan't generate a useful stub (domain too complex, custom evaluation logic needed), generate a custom benchmark with LLM assistance:- Read the project's main source files to understand the domain
- Identify the 2–3 most critical quality signals (latency, accuracy, throughput, precision/recall)
- Write a benchmark that measures those signals and outputs
{"composite": 0.0–1.0, ...} - Save to
benchmarks/eval_runner.{ext}matching the project language
-
Wire into eval.yaml:
benchmarks: - name: eval_runner command: python3 benchmarks/eval_runner.py full result_type: composite # parse composite field from JSON stdoutUse
result_type: exit_codefor frameworks (criterion, JMH, BenchmarkDotNet) that manage their own output.
Step 1: Run Rust CLI
Execute the structured evaluation via the Rust binary:
epic eval --json
This runs all enabled dimensions and outputs a JSON result. Capture the output.
If the CLI reports llm_judge: SKIPPED (no LLM available in CLI mode), proceed to Step 2 for LLM-as-judge. Otherwise, skip to Step 3.
Step 2: LLM-as-Judge (when llm_judge enabled)
If the quality dimension has llm_judge: true and CLI marked it SKIPPED:
-
Sample 3–5 changed files from the current branch:
git diff --name-only $(git merge-base HEAD main) -
For each sampled file, evaluate on a 1-10 rubric:
- Readability (naming, structure, flow)
- Correctness (logic, edge cases, error handling)
- DRY (no unjustified duplication)
- Security (no obvious vulnerabilities)
-
Average scores across files. Map to 0.0–1.0 scale.
-
Record results alongside CLI output.
Step 3: Load Baseline
cat $HARNESS_DIR/eval/baselines/latest.json
If no baseline exists, the current run BECOMES the first baseline. Save it:
epic eval --baseline-update
Report: "First baseline established. Future runs will compare against this."
Step 4: Synthesize Report
Combine CLI output + LLM-as-judge results into a single report:
## Eval Report
- Branch: {branch}
- Commit: {commit_short}
### Correctness: [PASS/WARN/FAIL] — score: {score}
- Tests: {passed}/{total} passing ({pass_rate}%)
- Mutation score: {mutation_score}% (if enabled)
- Delta vs baseline: {+/-delta}
### Performance: [PASS/WARN/FAIL] — score: {score} (if enabled)
- Avg latency: {latency}ms (delta: {+/-delta})
- Throughput: {throughput} (delta: {+/-delta})
### Quality: [PASS/WARN/FAIL] — score: {score}
- Lint errors: {count}
- LLM judge: {score}/10 (if enabled)
### Regression: [PASS/FAIL]
| Dimension | Baseline | Current | Delta | Verdict |
|-----------|----------|---------|-------|---------|
| correctness | {prev} | {cur} | {delta} | {pass/fail} |
| quality | {prev} | {cur} | {delta} | {pass/fail} |
### Overall: [PASS/WARN/FAIL] — {overall_score}
Step 5: Act
- All PASS + no regression: "Eval passed. Run
/shipto create a PR." - WARN: Show warnings. Ask whether to fix before shipping.
- FAIL or regression detected: List each failure with fix hint. "Fix with
/go, then re-run/eval."
Step 6: Save Results
epic eval --baseline-update # if user approves this as new baseline
Results auto-saved to $HARNESS_DIR/eval/results/EVAL-{timestamp}.json.
Anti-Rationalization
| Excuse | Rebuttal | What to do instead |
|---|---|---|
| "Tests pass, no need for eval" | Tests pass today but regress tomorrow without baselines | Run eval and establish a baseline |
| "Performance testing is premature" | Latency regressions are invisible until users complain | Enable performance dimension, run benchmarks now |
| "Mutation testing is too slow" | Slow mutation catches bugs fast tests miss | Run on changed modules only (--dimension correctness) |
| "LLM-as-judge is subjective" | Subjective beats absent — fixed rubric + averaging reduces variance | Use the 4-axis rubric, average across 3+ files |
| "We can add eval later" | Later never comes; regressions accumulate silently | Start with correctness+quality, add dimensions incrementally |
| "CI will catch regressions" | CI only catches build/test failures, not quality drift | Eval measures what CI misses: mutation score, LLM quality |
Evidence Required
-
epic eval --jsonoutput captured (all enabled dimensions scored) - Baseline comparison performed (or first baseline established)
- Each dimension has PASS/WARN/FAIL verdict
- No dimension regressed beyond threshold (or explicit user override)
- Results saved to
$HARNESS_DIR/eval/results/ - LLM-as-judge scores recorded (if enabled)
Red Flags
- Reporting PASS without actual
epic evaloutput - Skipping regression comparison "because it's the first run" (first run should ESTABLISH baseline)
- Reporting PASS with 0 test coverage
- Ignoring mutation score drops >5%
- Marking eval PASS when any dimension below minimum threshold
- Running eval on main branch instead of feature branch