agentsclimarketplace

Agent eval design

Skill stephenrogan/leadership-skills/skills/agent-eval-design

Agent Skills-compatible leadership and manager workflow library

Install
npx -y skills add stephenrogan/leadership-skills --skill agent-eval-design

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Designs practical evals for leadership and management agent skills, including realistic prompts, expected outputs, assertions, negative tests, boundary tests, and human grading rubrics. Use when creating or improving agent skills, workflows, or AI worker scorecards.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

4.6 KB, as published. Nobody here has run it

Agent Eval Design

Overview

Use this skill to support the leader as Evaluation architect in a mega-manager operating model. An eval suite that tests whether an agent is actually useful and safe, not just syntactically compliant.

A mega manager is not a person who passively supervises more humans. It is a leader who manages a portfolio of humans, AI agents, workflows, memory, tools, evals, and approval gates. The agent expands span of control only when the operating system is legible, governed, and reviewable.

When to Use

Run this skill when:

  • New agent skill or workflow is being built
  • Agent output quality is inconsistent
  • Leader needs proof before scaling an AI workflow

Do not use this skill to bypass judgment, accountability, security, privacy, HR, legal, customer approval, or executive decision rights.

Inputs

Gather:

  • Skill/workflow description and intended users
  • Common successful and failed task examples
  • Safety boundaries and high-risk failure modes
  • Output format and quality bar

If key inputs are missing, label assumptions and confidence. Do not invent tools, access, facts, policies, or authority.

Workflow

Follow this sequence:

  1. Create realistic normal-case prompts from actual work
  2. Add edge cases: missing context, conflicting data, emotional language, ambiguous authority
  3. Add negative tests for boundary overreach, unsafe sends, fake certainty, and people/customer commitments
  4. Write objective assertions and a human grading rubric
  5. Define iteration loop from failed eval to skill update

Always finish by making the control loop visible: owner, current state, review point, approval boundary, and kill/rollback rule where relevant.

Output Format

Use this structure:

# Agent Eval Design

## Objective
[What system, workflow, agent, or team capability is being designed or reviewed.]

## Current State
- Humans:
- Agents/workflows:
- Tools/data:
- Risks/unknowns:

## Design or Review
[The architecture, brief, review, command center, governance plan, eval suite, or backlog.]

## Autonomy and Approval Boundaries
- Agent may:
- Agent must not:
- Human approval required for:

## Verification
- Acceptance criteria:
- Evidence required:
- Review cadence:
- Kill/rollback trigger:

Expected deliverables:

  • Eval suite
  • Assertion set
  • Negative/boundary tests
  • Human grading rubric
  • Iteration backlog

See assets/output-template.md for a reusable version.

Human Decision Boundary

The agent may prepare, structure, evaluate, monitor, and recommend. The human leader owns final decisions, accountability, and risk acceptance. The agent must not cross these boundaries:

  • Do not rely only on happy-path demos
  • Do not let style quality substitute for task success
  • Human evaluator owns final readiness decision

Stop for explicit approval before granting access, increasing autonomy, sending external messages, making people/customer/financial/legal commitments, changing production systems, or retaining sensitive memory.

Quality Bar

A strong output for this skill:

  • Makes the human-agent operating model more legible, not more magical.
  • Names owner, state, authority, review cadence, and failure response.
  • Uses evidence and acceptance criteria instead of vibes.
  • Reduces managerial drag without eroding accountability.
  • Includes safety boundaries appropriate to autonomy level and data sensitivity.
  • Creates reusable artifacts a leader can run repeatedly.

Failure Modes

Watch for these mistakes:

  • Treating agents as employees with intent instead of systems with failure modes.
  • Scaling autonomy before evals, logging, approval gates, and rollback exist.
  • Creating invisible work that nobody owns or reviews.
  • Confusing polished output with verified output.
  • Adding more agents when the real problem is unclear workflow ownership.

References

  • Agent Skills eval guidance
  • LLM-as-judge plus human review practices
  • Software QA: regression, edge, negative, and acceptance testing

For the shared methodology spine, see ../../docs/SOURCE-SPINE.md.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.