Eval

Evaluation harness: $ARGUMENTS

Commands

Parse the first word of $ARGUMENTS to determine the command:

metrics — Show current session metrics

Collect and display metrics for the current session:

1. Count commits: git log --oneline --since="today" | wc -l
2. Count test results: check for recent vitest output or .eval/metrics/ entries
3. Token usage: check LangSmith state file if available
4. Pattern usage: check .dgm/fitness.json for patterns used this session
5. Session duration: check session start time from logs

Display as:

## Current Session Metrics

| Metric | Value | Baseline | Delta |
|--------|-------|----------|-------|
| Commits | 5 | 3.2 avg | +56% |
| Tests passing | 42/42 | 40/42 | +2 |

| Files changed | 12 | 8.5 avg | +41% |
| Patterns used | 7 | 5.3 avg | +32% |

baseline — Set or update baselines

1. Read the last N session metric snapshots from .eval/metrics/
2. Calculate averages for each metric
3. Write to .eval/baselines.json
4. Report what changed

compare — Compare sessions

Usage: compare --last N or compare --session <id>

1. Load metric snapshots from .eval/metrics/
2. Compare against baselines
3. Highlight regressions (metric dropped >10% below baseline)
4. Highlight improvements (metric improved >10% above baseline)

report — Generate weekly evaluation report

1. Load all metrics from the past 7 days
2. Calculate trends (improving, stable, declining)
3. Identify top improvements and top regressions
4. Generate recommendations based on trends

capture — Capture current session metrics

Write a metric snapshot to .eval/metrics/session-{timestamp}.json:

{
  "session_id": "<session id>",
  "timestamp": "<ISO 8601>",
  "branch": "<git branch>",
  "metrics": {
    "commits": 0,
    "tests_total": 0,
    "tests_passing": 0,
    "files_changed": 0,
    "patterns_referenced": 0,
    "assumptions_verified": 0,
    "escalations": 0,
    "spiral_detections": 0
  },
  "qualitative": {
    "session_focus": "<what the session was about>",
    "memory_usefulness": 0,
    "knowledge_gaps_found": []
  }
}

Notes

• If .eval/baselines.json doesn't exist, skip baseline comparisons and suggest running baseline
• Metric collection should be best-effort — missing data is noted, not an error
• Regressions trigger a structured escalation suggestion (not automatic action)