Reasoning rubric grader

Reasoning Rubric Grader

What this is (honestly)

This is a self-QA checklist, not a capability upgrade. It doesn't make a model smarter — it forces an explicit, evidence-based grading pass instead of a vague "does this look okay?" gut check, and it blocks delivery until the draft clears a bar you stated before you graded it. It works best on tasks with checkable dimensions (correctness, completeness, format compliance, source fidelity). It works poorly — and shouldn't be forced — on purely subjective or creative tasks where the rubric itself would just be guesswork dressed up as rigor.

When to trigger

• The task has explicit or implicit success criteria: required fields, format constraints, factual claims that must trace to a source, code that must not break existing behavior, word/length limits, etc.
• High-stakes or easy-to-get-subtly-wrong deliverables (data extraction, structured reports, decisions with hard constraints).
• Not for open-ended chat or tasks where "good" is purely a matter of taste — grading those numerically is theater, not rigor. Say so and skip it.

Steps

1. Build the rubric before grading — ideally before drafting. 3–6 dimensions, no more (more than that turns a cheap QA pass into a research project and defeats the point of keeping this bounded). For each dimension write:

   • A short name.
   • A one-line definition of what full marks looks like.
   • A weight, if some dimensions matter more than others (plain pass/fail is fine too). Derive dimensions from three sources: the task's explicit requirements, its implicit format constraints, and the known failure modes for this task type (for code: correctness, edge cases, security, no unrelated changes; for extraction: completeness, source fidelity, format; for a report: coverage, accuracy, actionability).

2. Draft the answer normally.

3. Grade the draft against the rubric, dimension by dimension. For every dimension give a score plus one line of evidence — a quote or specific pointer into the draft that justifies the score. A score with no evidence doesn't count; it's the same vague gut check the rubric exists to replace.

4. Compute a verdict against a threshold you set upfront (default: every dimension ≥ 3/5 AND weighted average ≥ 4/5; or, simpler, "every must-pass dimension = PASS").

5. If it fails: revise only the failing dimension(s), then re-grade. Cap at 2 revision rounds.

6. If it still fails after the cap: deliver the best version you have AND say explicitly which dimension didn't clear the bar and why. Never silently rubber-stamp a failing draft to end the loop.

7. Deliver. Keep the scorecard internal by default; surface it to the user only if they asked for transparency, or if step 6 fired and you're flagging a gap.

Inputs

• The task itself (what "done" needs to include).
• Optional: user-supplied constraints or criteria to fold into the rubric.
• Optional: pass-threshold override (default above).
• Optional: max revision rounds override (default 2).

Output format

Internal scorecard (show only on request or on failure):

Dimension	Weight	Score /5	Evidence	Verdict
...	...	...	quote/pointer into the draft	PASS/FAIL

Followed by one verdict line — PASS (weighted X/5) or FAIL on <dimension>: <why> — and then the delivered answer.

Guardrails

• Max 6 dimensions, max 2 revision rounds — this is a cheap gate, not a second draft process.
• No score without pointed-to evidence in the draft.
• If the task is ungradable (pure opinion/creative), say so and skip numeric grading rather than fabricate a rubric to look rigorous.

Worked example

Task: "From this email thread, extract: decision maker's name, role, company, and whether they're currently raising."

Rubric

Dimension	Weight	Definition of 5/5
Completeness	2	All 4 fields present, or explicitly marked "not found"
Source fidelity	2	Every value is directly traceable to the thread — no inference presented as fact
Format	1	Output matches the requested field order/labels

Draft: "Name: Marta Solà. Role: CFO. Company: FirstStone. Raising: yes — thread says 'we're closing our seed in Q3'."

Grading

• Completeness: 5/5 — all 4 fields present.
• Source fidelity: 1/5 — "Company: FirstStone" never appears anywhere in the thread; it's a hallucinated fill-in, not an extraction.
• Format: 5/5 — matches requested order/labels.

Weighted: (2×5 + 2×1 + 1×5) / 5 = 3.4/5, and Source fidelity < 3 → FAIL.

Revision: change to "Company: (not mentioned in thread)". Re-grade: Source fidelity → 5/5 (correctly marked absent instead of invented). Weighted 5/5, all dimensions ≥ 3 → PASS. Deliver.

Related techniques

Pairs well with other bounded reasoning-harness techniques for cheap models: plan-then-execute, self-consistency (sample + vote), reflexion (self-critique), adversarial verification. This skill is the "quality gate" step — it decides whether the output of those other steps is good enough to ship.