Ai forge judge

AI Forge Judge

Evaluate any LLM-consumed prompt against quality standards, focused on knowledge delta, instruction clarity, and practical usability.

---

Core Philosophy

Good Prompt = Expert-only Knowledge − What Claude Already Knows

Restating defaults is token waste.

Three Types of Knowledge

Type	Definition	Treatment
Expert	Claude genuinely doesn't know this	Must keep — this is the value
Activation	Claude knows but may not think of	Keep if brief — serves as reminder
Redundant	Claude definitely knows this	Delete — wastes tokens

Good prompt: >70% Expert, <20% Activation, <10% Redundant.

Evaluation Dimensions

Dimensions are grouped. Universal dimensions always apply. Type-specific modules apply based on what the prompt is. Multiple groups can apply to a single prompt.

Final grade = total score / total applicable points (as %)

Grade	%	Meaning
A	90%+	Excellent — production-ready
B	80–89%	Good — minor improvements needed
C	70–79%	Adequate — clear improvement path
D	60–69%	Below average — significant issues
F	<60%	Poor — needs fundamental redesign

Group U: Universal (80 pts) — always scored

MANDATORY — READ references/universal-dimensions.md

ID	Dimension	Pts
U1	Knowledge/Instruction Delta	20
U2	Mindset + Procedures	15
U3	Constraint Quality	15
U4	Freedom Calibration	15
U5	Practical Usability	15

Group S: Skill Module (40 pts) — SKILL.md targets only

MANDATORY — READ references/skill-dimensions.md

ID	Dimension	Pts
S1	Specification Compliance	15
S2	Progressive Disclosure	15
S3	Pattern Recognition	10

Group C: Agent / System Prompt Module (40 pts) — agent definitions and system prompts

MANDATORY — READ references/agent-dimensions.md

ID	Dimension	Pts
C1	Behavioral Clarity	15
C2	Scope Definition	15
C3	Structural Organization	10

Group B: Bash/Shell Module (30 pts) — prompts that contain shell/CLI guidance

MANDATORY — READ references/bash-dimensions.md

ID	Dimension	Pts
B1	Rule Specificity & WHY	10
B2	Anti-Pattern Coverage	10
B3	Scope & Exceptions	10

---

Evaluation Protocol

Evaluator's Lens: Before reading the target, adopt this question: "Does Claude already know this?" — every section gets marked [E], [A], or [R] before scoring begins.

Step 0: Detect Prompt Type

Read the target and identify which groups apply:

[ ] Is it a SKILL.md file?              → Score U + S. Do NOT load agent-dimensions.md or bash-dimensions.md.
MANDATORY: Load agentskills spec from references/agentskills-spec.md before scoring S1.
[ ] Is it an agent definition (.agent.md)? → Score U + C. Do NOT load skill-dimensions.md.
[ ] Is it a system prompt / CLAUDE.md?  → Score U + C. Do NOT load skill-dimensions.md.
[ ] Does it contain bash/shell rules?   → Also score B. Load bash-dimensions.md.
[ ] Is it something else?               → Score U only. Do NOT load any type-specific reference.

Multiple groups can apply (e.g. a SKILL.md with bash guidance → U + S + B).

Edge cases:

• SKILL.md that also contains bash guidance → U + S + B
• A referenced sub-file (e.g. references/bash.md, not a root prompt) → U only; note "sub-file, not root prompt" in report
• Ambiguous type (could be agent or skill) → score both C and S groups; note the ambiguity in the Summary
• Target <10 lines → Score U only; note "Too brief for full dimensional analysis — expand before re-evaluation"

Spec Reference

The agentskills.io specification is bundled at references/agentskills-spec.md. If you need the latest version, WebFetch https://agentskills.io/specification — but the bundled copy is the baseline for scoring.

Only load when the target is a SKILL.md (S1 scoring). Skip entirely for agent / system prompt / other evaluations.

Step 1: First Pass — Knowledge Delta Scan

Read completely. Mark each section [E] Expert | [A] Activation | [R] Redundant. Calculate E:A:R ratio — target >70% Expert.

If a section's classification is ambiguous (could be E or A), default to A. Never default to E — that inflates scores.

Step 2: Structure Analysis

Note prompt type(s), applicable groups, length, reference files, and loading/trigger mechanisms.

Step 3: Score Each Applicable Dimension

Before opening the rubric, ask: what does this section assume Claude doesn't already know?

Load the reference file for each applicable group. For each dimension: find specific evidence, assign score with one-line justification, note improvements if score < max.

If a reference file cannot be read, halt and report: [ERROR] Cannot score Group X — reference file not found: <path>. Do not proceed with that group.

Step 4: Calculate Score & Grade

Grade = (sum of scored dimensions) / (sum of applicable maxes) → apply grade scale.

Step 5: Generate Report

MANDATORY — READ references/report-template.md for the exact report structure. Follow it precisely.

---

Common Failure Patterns

MANDATORY — READ references/failure-patterns.md

---

Extending ai-forge-judge

To add a new evaluation group, MANDATORY — READ references/extending-groups.md before proposing any new group. Do NOT load this file during a normal evaluation run.

---

Self-Application

ai-forge-judge can and should evaluate itself. The criteria must be self-consistent — if ai-forge-judge can't score well against its own rubric, the rubric is wrong.

Applicable groups: ai-forge-judge is a SKILL.md with no bash guidance → U + S (120 pts max).

Expected score: ≥B (80%+, ≥96/120). A score below B indicates the rubric has drifted from its own standards.

---

NEVER Do When Evaluating

• NEVER give high scores just because it "looks professional" or is well-formatted — formatting is cheap; rewarding it masks content gaps INSTEAD: Score content for expert knowledge density, not visual polish.
• NEVER ignore token waste — redundant content dilutes expert signal INSTEAD: Deduct from U1 consistently regardless of overall quality; note the specific redundant lines.
• NEVER let length impress you — a 500-line prompt with 80% activation content is worse than a 50-line one with pure expert knowledge INSTEAD: Measure the Expert:Activation:Redundant ratio.
• NEVER skip mentally testing decision trees — plausible-looking trees often have unreachable branches INSTEAD: Trace each branch to verify it terminates with a clear action.
• NEVER forgive explaining basics with "but it provides helpful context" INSTEAD: Mark the section [R] and deduct from U1.
• NEVER treat the presence of a NEVER list as evidence of quality, or its absence as a defect INSTEAD: Ask whether the domain has recurring failure modes and whether they're addressed in any form. Score the justification. A constraint carried by explained reasoning beats the same constraint asserted as a prohibition, and a wall of low-value NEVERs dilutes the ones that matter.
• NEVER undervalue the description field for Skills — it is the only thing the agent sees before deciding whether to load INSTEAD: Score S1 harshly for vague or keyword-poor descriptions.
• NEVER compare percentage scores across evaluations without checking which groups were scored INSTEAD: Always note the denominator in the report.
• NEVER place Numbered Improvements before Detailed Analysis — the reader needs context before recommendations INSTEAD: Always write Detailed Analysis first, then Numbered Improvements immediately after.