Judge

judge

Backs subagent output evaluation across all /shannon:dispatch* commands. Implements the two-stage meta-judge → LLM-as-judge pattern with structured YAML verdicts, hidden-threshold bias mitigation, and integration with Shannon's iron-rules dimensions.

Two-stage pipeline

Stage 1 — meta-judge (delegated to agents/meta-judge.md):

1. Receives the original task, context, artifact type.
2. Generates an evaluation specification YAML containing rubrics, checklists, scoring criteria, forbidden patterns, and evidence requirements.
3. Output is a single block of YAML, written verbatim to .shannon/state/rubric-<run-id>.yaml.
4. Pass-threshold is NEVER included in the YAML. Judges must remain blind to the threshold to prevent bias.

Stage 2 — judge (this skill):

1. Receives the meta-judge YAML verbatim, the artifact under evaluation, and the original task.
2. Applies the rubric mechanically — score each criterion with cited evidence.
3. Emit a structured YAML verdict header at the START of the response.
4. The orchestrator parses only the header, never reads the full report.

Single-output mode (default)

For a single artifact (post-implementation evaluation):

1. Read subagent output + acceptance criteria from spawn context.
2. Apply the meta-judge YAML rubric across all defined dimensions plus Shannon's iron-rules dimension (always present, regardless of meta-judge content):
   • Completeness — covers all acceptance criteria.
   • Correctness — claims are accurate; code compiles; outputs match expected.
   • Quality — style match, citation density, KISS adherence.
   • Iron rules — no mocks, no test files, no fabricated evidence.
3. Self-verify: re-read the artifact and confirm each score with quoted evidence. Adjust scores if self-verification reveals miscalibration.
4. Emit verdict header.

Structured YAML verdict header

---
VERDICT: PASS | FAIL | NEEDS_REVISION
SCORE: X.X/5.0
CRITERIA:
  - {criterion_1}: {X.X}/5.0
  - {criterion_2}: {X.X}/5.0
  - iron_rules: {PASS | FAIL}
ISSUES:
  - {issue 1 with file:line citation}
  - {issue 2 with file:line citation}
IMPROVEMENTS:
  - {suggestion 1}
  - {suggestion 2}
SELF_VERIFICATION_ADJUSTMENTS:
  - {criterion that was rescored after self-verify, with reason}
---

## Detailed Evaluation
{Per-criterion analysis with quoted evidence per the meta-judge YAML's rubric}

The orchestrator parses ONLY this header. The detailed body is for the user's review, never re-read by the orchestrator (avoids context pollution).

Scoring interpretation

Score Range	Verdict	Interpretation	Recommendation
4.50 - 5.00	EXCELLENT	Exceeds expectations	Ready as-is
4.00 - 4.49	GOOD	Meets professional standards	Minor improvements optional
3.50 - 3.99	ACCEPTABLE	Adequate, room for improvement	Improvements recommended
3.00 - 3.49	NEEDS IMPROVEMENT	Below standard	Address issues before use
1.00 - 2.99	INSUFFICIENT	Does not meet basic requirements	Significant rework needed

Iron-rules dimension is binary (PASS / FAIL) regardless of numeric score. A FAIL on iron-rules forces VERDICT: FAIL on the whole artifact, even if numeric score is high. This is Shannon's distinctive guarantee against fabricated evidence and mocks.

Soft-pass rule

The orchestrator (NOT the judge) applies a soft-pass when:

SCORE >= 3.0 AND all ISSUES are low-priority AND iron_rules == PASS
→ Treat as PASS. Mark complete. Surface IMPROVEMENTS as optional.

The judge does not know this rule (no threshold disclosure). The judge reports raw scores; the orchestrator decides.

Debate mode (escalation for ties or stakes)

When a single judge produces ambiguous output (score near boundary, contested ISSUES, or competitive ranking ties), escalate to multi-round debate:

1. Spawn 2-3 independent judges in parallel via Task. Each receives the SAME meta-judge YAML.
2. Each emits an independent verdict header.
3. If overall scores disagree by > 0.5 points OR any criterion disagrees by > 1.0 point → spawn debate round.
4. In each debate round, judges read peers' reports from the filesystem, defend or revise positions grounded in the meta-judge YAML, append ## Debate Round R to their own report. Max 3 rounds.
5. Final verdict: average of converged scores; SPLIT if no convergence after 3 rounds.

Debate is delegated to the judge-with-debate skill for the iteration mechanics. This skill handles single-output evaluation; debate is a layered capability.

No-self-judging rule

A judge that helped produce an artifact may NOT also judge it. The orchestrator enforces this by tracking subagent identities and refusing to dispatch a judge that overlaps. Shannon's audit trail records the judge identity per verdict.

When to use

• Every subagent output in dispatch-parallel (default sequential dispatch).
• Competitive mode tie-breaking (delegate to debate mode).
• Pre-meta-judge synthesis in parallel mode (one judge per target).
• Standalone artifact evaluation via /shannon:reflect --mode judge.

When NOT to use

• Trivial outputs where success is binary and a hard gate already proves it (e.g. compiled successfully). Use the gate result directly.
• Re-judging an artifact already judged in this run (idempotent — re-judging poisons the audit).

Iron rules

• Judge does NOT author. Judge reads + emits verdict.
• No "good enough" — PASS / FAIL / NEEDS_REVISION verdict required.
• Cited reasoning per dimension. No vague PASSes.
• No self-judging — judge that helped produce an output cannot judge it.
• Pass-threshold is NEVER disclosed to the judge (bias mitigation).
• Meta-judge YAML is passed verbatim — no edits, no truncation, no summary.
• Iron-rules dimension is mandatory regardless of meta-judge YAML content. A FAIL on iron-rules trumps any numeric score.

Related Skills

• agents/meta-judge.md — the rubric-generator that runs BEFORE this skill.
• judge-with-debate — multi-round debate engine for SPLIT / contested verdicts.
• critique — adversarial framing (BLOCKING / HIGH / MEDIUM / LOW severity), distinct from this skill's scored evaluation.
• consensus-engine — multi-validator synthesis (5-state synthesis table); uses this skill as the underlying judge.
• judge — quorum gate with file-ownership rules; uses this skill as the underlying judge per oracle.
• tree-of-thoughts — uses this skill in two phases (pruning judges + evaluation judges).
• dispatch-parallel — orchestrator that dispatches one instance of this skill per target.

Cross-references

• agents/meta-judge.md — Stage 1 rubric generator.
• .shannon/state/rubric-<run-id>.yaml — persisted meta-judge YAML for audit.
• core/SUBAGENT_PATTERNS.md — design rationale.

---

Absorbed (Phase 1 aggressive merge)

Skills merged into this canonical survivor during Shannon v0.1.0 Phase 1 curation. Content preserved for Phase 2 canonical-merge work.

Absorbed from critique

critique

Backs /shannon:reflect --mode critique. Spawns the critic agent in isolated context with meta-judge-tailored criteria per artifact type. Distinct from judge (scored evaluation) — critique is adversarial framing, one-shot, severity-classified, non-iterative.

Two-stage pipeline

Stage 1 — meta-judge tailoring (delegated to agents/meta-judge.md):

• Receives the artifact type (PRD vs code diff vs evidence vs plan) and original task.
• Returns a critique-criteria YAML — what to attack, what to scrutinize, where the artifact is most likely to fail.
• Different artifact types get different attack surfaces. A PRD critique looks for ambiguity, contradiction, untestable acceptance criteria; a code diff critique looks for regressions, missing edge cases, performance cliffs; an evidence critique looks for fabrication, citation gaps, mock contamination.

Stage 2 — critic agent (this skill):

• Receives the meta-judge YAML, the artifact, the original task.
• Runs adversarial scan. Emits findings classified BLOCKING / HIGH / MEDIUM / LOW with cited file:line.

Behavior contract

1. Identify the artifact under critique (plan, PRD, code diff, evidence package). Determine artifact type.
2. Dispatch meta-judge to generate critique-criteria YAML tailored to artifact type.
3. Spawn critic agent via Task. Critic owns its output directory (reports/critique-<run-id>/).
4. Critic emits findings classified BLOCKING / HIGH / MEDIUM / LOW with cited file:line.
5. Critic emits a structured YAML header at the START of its response (for orchestrator parsing).
6. Aggregate critic output; surface to user via critique.md.

Severity classification

Severity	Meaning	Action Required
BLOCKING	Defect that prevents shipping (security hole, data corruption, regression, mock contamination, iron-rule violation)	Must fix before any forward motion
HIGH	Significant defect that should be fixed before merge (missing edge case, untested critical path, ambiguous spec)	Fix in this iteration
MEDIUM	Defect that should be fixed soon (style inconsistency, weak test, unclear name)	Fix in this or next iteration
LOW	Cosmetic or stylistic improvement (doc clarity, comment polish)	Optional

BLOCKING is sharper than scores. A score of 4.2/5.0 with one BLOCKING finding is NOT a pass. Severity classification is more actionable than numeric scores for adversarial review.

Structured YAML output header

---
ARTIFACT: {path or identifier}
ARTIFACT_TYPE: {plan | prd | code-diff | evidence | other}
META_JUDGE_RUBRIC: .shannon/state/critique-rubric-<run-id>.yaml
SUMMARY_COUNTS:
  BLOCKING: {N}
  HIGH: {N}
  MEDIUM: {N}
  LOW: {N}
VERDICT: {BLOCKING_FOUND | HIGH_FOUND | CLEAN}
---

Output (full report)

# Critique — <artifact>

## BLOCKING
- <finding> [<file>:<line>] — <one-sentence explanation>
...

## HIGH
- <finding> [<file>:<line>] — <one-sentence explanation>
...

## MEDIUM
- <finding> [<file>:<line>] — <one-sentence explanation>
...

## LOW
- <finding> [<file>:<line>] — <one-sentence explanation>
...

## Patterns observed
{Any systemic issues — recurring patterns across findings}

Anti-patterns table

Pattern	Why It's Wrong	Do This Instead
Softening a BLOCKING to HIGH "to keep momentum"	Defeats the entire purpose of critique	Keep BLOCKING. Document the consequences. Let the author decide.
Critic iterates with author back-and-forth	Critique is one-shot adversarial; iteration is for judge + retry	Critic emits findings once. Author addresses them separately.
Vague findings ("this code is messy")	Not actionable	Cite file:line with one-sentence "why this is a defect"
Critique without artifact-tailored criteria	Generic critique misses domain-specific failure modes	Always run meta-judge first to tailor critique criteria
Critique self-authored artifact	No-self-judging applies to critique too	Spawn an isolated critic; identity-disjoint from author
Treating critique as a quality gate	Critique surfaces findings; doesn't decide PASS/FAIL	Pair critique with evidence-gate or completion-gate

When to use

• Pre-merge code review (before peer review).
• Pre-execution plan review (lighter than oracle quorum — single critic, not 3-oracle vote).
• After a near-miss to harden the next iteration.
• PRD review before plan-author.
• Evidence package audit before submitting for oracle quorum.

When NOT to use

• Quick sanity check → use /shannon:reflect self-mode (no critic agent).
• Already-reviewed artifact → don't re-critique what oracle has gated.
• Hard PASS/FAIL gate → use evidence-gate or completion-gate; critique surfaces findings only.

Iron rules

• Critique is one-shot. Critic does not iterate with author.
• Findings cite file:line. Directory references do not count.
• No softening — BLOCKING stays BLOCKING.
• No self-critiquing — critic identity is disjoint from author.
• Meta-judge runs BEFORE critic to tailor criteria per artifact type.
• Critic emits structured YAML header for orchestrator parsing.

Related Skills

• agents/meta-judge.md — tailors critique criteria per artifact type before the critic runs.
• judge — scored evaluation; this skill is adversarial-framing alternative.
• judge — heavier quorum gate (3 oracles); critique is single critic.
• evidence-gate, completion-gate — actual PASS/FAIL gates; critique surfaces findings, doesn't gate.
• plan-author — author↔red-teamer iteration; uses critique as the red-team voice.

Cross-references

• agents/critic.md — the adversarial agent invoked here.
• agents/meta-judge.md — tailors critique criteria.
• .shannon/state/critique-rubric-<run-id>.yaml — persisted critique criteria.

Absorbed from judge

judge

Quorum gate. Invoked at two points: BEFORE execution (plan review) AND AFTER execution (evidence audit). Single-round quorum is the default; multi-round debate is the escalation path on SPLIT or borderline MAJORITY.

Behavior contract

Pre-execution (plan-review)

1. Dispatch meta-judge ONCE (via agents/meta-judge.md) to generate the plan-review rubric YAML. Stored at .shannon/state/oracle-rubric-<run-id>.yaml.
2. Spawn 3 oracle agents in parallel, isolated contexts (SINGLE assistant response, 3 Task tool calls). Each receives:
   • The plan + codebase-analysis + library-docs-fetch artifacts.
   • The meta-judge YAML verbatim.
   • Its assigned output path: e2e-evidence/<run-id>/oracle-quorum/oracle-<N>/verdict.md.
3. Each oracle independently emits a structured YAML verdict header:

   ---
   VERDICT: APPROVE | REFUSE
   CRITICAL_BLOCKERS: [count]
   CITED_BLOCKER:
     - file: <path>
       line: <range>
       reason: <one-sentence>
   SCORES:
     - {criterion}: {X.X}/5.0
   SUMMARY: <one paragraph>
   ---
   

4. Aggregate verdict headers. Apply quorum rule:
   • ≥2 APPROVE AND 0 unresolved critical blockers → PASS; pipeline proceeds.
   • Otherwise → check for escalation conditions (see Debate escalation).
5. On final REFUSE → invoke refusal-discipline to emit REFUSAL.md; halt pipeline.

Post-execution (evidence-audit)

1. Dispatch meta-judge ONCE to generate the evidence-audit rubric YAML.
2. Spawn 3 oracle agents in parallel. Each reads cited evidence per MSC and votes APPROVE / REFUSE.
3. Aggregate via same quorum rule. Same escalation behavior.

Debate escalation

Default = single-round quorum. Escalate to multi-round debate when:

• SPLIT — votes are 1 APPROVE / 1 REFUSE / 1 APPROVE (or similar non-2-aligned) and critical-blockers != 0.
• Borderline MAJORITY — 2 APPROVE but with CITED_BLOCKER entries that disagree about severity (one oracle calls a finding critical, another calls it minor).

Escalation procedure (delegated to judge-with-debate):

1. Hand off to judge-with-debate skill with:
   • The 3 oracle reports as initial judges' reports (.specs/reports/oracle-<run-id>.[1|2|3].md).
   • The original artifact (plan or evidence).
   • The same meta-judge YAML (no new rubric generation).
2. Debate runs up to 3 rounds. Oracles read peers' reports DIRECTLY FROM THE FILESYSTEM — orchestrator does NOT mediate.
3. Convergence thresholds (from judge-with-debate): overall scores within 0.5 points AND every criterion within 1.0 point.
4. Filesystem as debate medium — oracles append ## Debate Round R to their own report files. Reports are append-only; debate history is preserved.
5. After convergence (or 3 rounds, whichever first):
   • Converged APPROVE majority → PASS.
   • Converged REFUSE majority → REFUSE; invoke refusal-discipline.
   • Still SPLIT after 3 rounds → emit SPLIT verdict (low confidence). User decision required.

Adaptive strategy selection

After single-round quorum AND optional debate, the result drives strategy:

Outcome	Strategy	Action
Unanimous APPROVE	PASS	Pipeline proceeds. Cherry-pick improvements from oracle IMPROVEMENTS lists.
Majority APPROVE, no blockers	PASS	Pipeline proceeds with surfaced IMPROVEMENTS as optional.
SPLIT after debate	NEEDS_USER_DECISION	Surface all 3 reports + debate history; user decides.
Unanimous REFUSE	REFUSE	Invoke refusal-discipline. Halt.
Critical blocker present	REFUSE	Even if score-majority APPROVE, ANY unresolved critical blocker forces REFUSE.

File ownership (LOAD-BEARING)

• Coordinator: writes NOTHING.
• Oracle-N: writes ONLY e2e-evidence/<run-id>/oracle-quorum/oracle-<N>/verdict.md (append-only across debate rounds).
• Synthesizer: writes ONLY e2e-evidence/<run-id>/oracle-quorum/report.md.
• Meta-judge: writes ONLY .shannon/state/oracle-rubric-<run-id>.yaml.

A cross-write invalidates the run. Append-only is enforced by the synthesizer's pre-write check.

Per-criterion score table (consensus report)

After quorum (and optional debate), synthesizer emits:

# Oracle Quorum Report — <run-id>

## Per-criterion scores

| Criterion | Oracle 1 | Oracle 2 | Oracle 3 | Consensus | Confidence |
|-----------|----------|----------|----------|-----------|------------|
| {criterion_1} | {X.X} | {X.X} | {X.X} | {avg} | {HIGH|MEDIUM|LOW} |
| ... | ... | ... | ... | ... | ... |

## Verdict
- Final: {APPROVE | REFUSE | SPLIT}
- Critical Blockers: {count}
- Debate Rounds: {0-3}
- Confidence: {HIGH | MEDIUM | LOW}

## Cited Blockers (if any)
- {file:line} — {reason} — first surfaced by Oracle {N}

## Improvements (optional)
- {improvement 1}
- {improvement 2}

When to use

• the pre-execution review step in /shannon:cook (Task: critic runs review before executor spawns) (pre-execution plan review).
• the post-execution review step in /shannon:cook (post-execution evidence audit).
• Standalone audit via /shannon:audit-completion.
• Major-release gating.

When NOT to use

• Routine /shannon:cook runs (lightweight) — only forge invokes oracle quorum.
• Single artifact evaluation → use judge (single judge, no quorum).
• Adversarial finding surfacing → use critique (single critic, no quorum).

Iron rules

• Distinct from reviewer. Oracle = pre-execution gate OR final evidence audit. Reviewer = post-implementation code/security/perf review. Both can exist for the same run.
• No self-review. Oracle that produced an artifact may not also review it.
• Quorum is hard. 2/3 APPROVE WITH 0 critical blockers, OR refusal. No "almost" pass.
• Citations specific. Directory references do not count; file path + line range where relevant.
• Single-round quorum is the default; debate is the escalation path — do not always debate (cost), do not refuse to debate when needed (quality).
• Filesystem is the debate medium. Coordinator does NOT relay oracle arguments. Oracles read peers' reports directly.
• Append-only on debate. Each oracle appends ## Debate Round R to its own file; never overwrites previous rounds.
• Threshold hidden from oracles (delegated via meta-judge YAML — no pass_threshold field).

Related Skills

• agents/meta-judge.md — generates judge rubric before quorum.
• judge-with-debate — engine invoked for multi-round escalation.
• consensus-engine — alternative validator-quorum primitive (5-state synthesis table); judge is plan/evidence-specific.
• judge — single-judge primitive; judge wraps 3 instances.
• critique — single adversarial critic; lighter than oracle quorum.
• refusal-discipline — invoked on final REFUSE to emit REFUSAL.md.

Cross-references

• agents/oracle.md — the oracle agent.
• agents/meta-judge.md — rubric generator.
• core/SUBAGENT_PATTERNS.md — quorum vs consensus vs debate design rationale.