Llm as judge loop

Skill kjuhwa/skills-hub/skills/llm-agents/llm-as-judge-loop

Self-correcting knowledge corpus for Claude Code — 9 stable shape clusters, bias-correction pipeline baked into contribution flow. 47 papers, 45 techniques, 1.1k skills.

Install

npx -y skills add kjuhwa/skills-hub --skill llm-as-judge-loop

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Run an agent in a loop with an evaluator agent that provides feedback until the output meets a quality threshold.

SKILL.md

2.8 KB, as published. Nobody here has run it

llm-as-judge-loop

Alternate between a generator agent and an evaluator agent in a while loop. The evaluator provides structured feedback (pass/fail/needs_improvement); loop continues until the score is "pass" or a max round limit is hit.

When to apply

Creative generation, code writing, or any task where quality is subjective and iterative refinement is valuable. Replaces manual review with automated LLM quality gates.

Core snippet

import asyncio
from dataclasses import dataclass
from typing import Literal
from agents import Agent, ItemHelpers, Runner, TResponseInputItem, trace

story_outline_generator = Agent(
    name="story_outline_generator",
    instructions="Generate a short story outline. Use feedback to improve if provided.",
)

@dataclass
class EvaluationFeedback:
    feedback: str
    score: Literal["pass", "needs_improvement", "fail"]

evaluator = Agent[None](
    name="evaluator",
    instructions="Evaluate the story outline. Provide feedback until it reaches a pass score.",
    output_type=EvaluationFeedback,
)

async def main():
    input_items: list[TResponseInputItem] = [{"content": "A detective story in space.", "role": "user"}]
    latest_outline: str | None = None

    with trace("LLM as a judge"):
        while True:
            outline_result = await Runner.run(story_outline_generator, input_items)
            latest_outline = ItemHelpers.text_message_outputs(outline_result.new_items)
            
            eval_result = await Runner.run(evaluator, latest_outline)
            evaluation = eval_result.final_output_as(EvaluationFeedback)
            
            if evaluation.score == "pass":
                break
            
            # Feed feedback back into generator input
            input_items = outline_result.to_input_list() + [
                {"content": f"Feedback: {evaluation.feedback}", "role": "user"}
            ]

    print(latest_outline)

asyncio.run(main())

Key notes

Cap loop iterations (max_rounds) to avoid infinite loops
Use result.to_input_list() to carry full conversation history into the next generator call
The evaluator uses structured output for type-safe pass/fail branching
Wrap in trace() to see all rounds in the OpenAI Traces dashboard