Managing ai evals

Skill pvnarp/harness-agentic-skills/skills/harness/managing-ai-evals

Manage the Harness AI Evals control plane via MCP. Create and version evaluation datasets (managed JSONL or git-backed), define metrics and metric sets with thresholds, register LLM models and invocation targets (prompt, agent, precomputed), wire datasets + targets + metric sets into evaluations, run them, compare runs, rescore historical runs, and group evaluations into suites with pass strategies and cron schedules. Use when asked to set up an LLM eval, benchmark a prompt or agent, build a regression suite, calibrate metrics, score model output, or run an eval suite on a schedule. Trigger phrases: AI eval, LLM eval, evaluation dataset, metric set, eval target, eval run, eval suite, rescore, calibrate metrics, prompt benchmark, agent benchmark.From its SKILL.md

Install

npx -y skills add pvnarp/harness-agentic-skills --skill managing-ai-evals

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as Apache-2.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

15.8 KB, as published. Nobody here has run it

Manage AI Evals

What this skill uses

Harness MCP tools: harness:harness_create, harness:harness_describe, harness:harness_execute, harness:harness_get, harness:harness_list, harness:harness_update Resource types: eval_dataset, eval_dataset_item, eval_metric, eval_metric_set, eval_model, eval_run, eval_run_item, eval_suite, eval_suite_run, eval_target, evaluation References: references/eval-yaml-import-export.md

See also: references/eval-yaml-import-export.md - eval YAML import/export (suite + evaluation YAML shapes, dry_run, git_source).

Drive the Harness AI Evals platform: datasets, metrics, models, targets, evaluations, runs, suites, annotations, analytics, registry items. All resources are project-scoped.

Schema discovery first. Field names below are pulled from the live MCP registry as of authoring. If unsure, run harness:harness_describe(resource_type='<eval_*>') for the current bodySchema. The eval surface evolves quickly.

Resource map

Resource type	Purpose
`eval_dataset` / `eval_dataset_item`	JSONL or git-backed test data, plus individual rows
`eval_metric` / `eval_metric_set` / `eval_metric_set_entry`	Metrics, grouped sets, and set membership
`eval_model`	Registered LLM models for runs and judge passes
`eval_target`	Invocation target: `prompt`, `agent`, or `precomputed` outputs
`evaluation`	The wiring of dataset + target + metric set
`eval_run` / `eval_run_item` / `eval_run_by_eval`	Runs and per-item scores; shortcut to filter runs by eval
`eval_suite` / `eval_suite_evaluation` / `eval_suite_run`	Multi-eval suites with pass strategies and schedules
`eval_annotation`	Trace-level human or automated annotations
`eval_analytics`	Project-level analytics summary (singleton)
`eval_git_settings`	Project git sync settings (singleton)
`eval_registry_item`	GenAI registry: prompts, agents, MCP tools, skills

YAML import shortcut

For batch authoring, the fastest path is evaluation + import_yaml (creates target, dataset, metric set, and eval in one call), or eval_suite + import_yaml (creates a suite with member evaluations). Use dry_run: true first to validate. See Step 8: YAML import / export below.

Instructions

Step 1: Datasets

List datasets:

Call MCP tool: harness:harness_list
Parameters:
  resource_type: "eval_dataset"
  org_id: "<org>"
  project_id: "<project>"
  filters:
    search: "<keyword>"

Create. name and identifier are required. items is optional inline-load (must be empty when storage_type='git'):

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_dataset"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "Support QA v1"
    identifier: "support_qa_v1"
    description: "QA dataset for support agent"
    storage_type: "managed"     # or "git" with git_source
    items: []                    # optional inline load

Bulk-load items via eval_dataset_item.bulk_upsert:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_dataset_item"
  action: "bulk_upsert"
  params:
    dataset_id: "<dataset-uuid>"
  body:
    items:
      - input: { question: "What is our SLA?" }
        expected: { answer: "99.9% uptime" }
      - input: { question: "How do I reset my password?" }
        expected: { answer: "Use the password reset link" }

Generate synthetic items asynchronously. strategy, count, and model_id are required:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_dataset"
  action: "generate"
  resource_id: "<dataset-uuid>"
  body:
    strategy: "use_case"          # use_case | rephrase | adversarial | complexity_ladder
    count: 50
    model_id: "<eval_model-uuid>"
    description: "Diverse user questions about platform billing."

Then poll with action: "poll_generate" (pass dataset_id and job_id from the generate response).

Step 2: Metrics and metric sets

Create a metric. Required: name, type. kind and config describe what the metric actually measures. There is no top-level judge_model_id or prompt on eval_metric itself - judge config goes in config:

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_metric"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "answer_correctness"
    type: "llm"                  # heuristic | llm | embedding | code | composite
    kind: "exact_match"          # harness-evals metric kind
    description: "Match expected answer with rubric"
    config:
      model_id: "<eval_model-uuid>"
      rubric: "Score 0-5 on factual correctness"
    default_threshold: 0.8

Create a metric set. judge_model_id lives on the set (default judge for LLM metrics in the set). Initial entries via entries:

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_metric_set"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "support-quality"
    description: "Quality bar for the support assistant"
    judge_model_id: "<eval_model-uuid>"
    entries:
      - metric_id: "<metric-uuid-1>"
        threshold: 0.85
      - metric_id: "<metric-uuid-2>"
        threshold: 4.0

Replace the metric list later via replace_metrics (PUT). Body is a raw JSON array of AddMetricSetEntryRequest objects:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_metric_set"
  action: "replace_metrics"
  resource_id: "<set-uuid>"
  body: [
    { metric_id: "<id-1>", threshold: 0.9 },
    { metric_id: "<id-2>", threshold: 4.5 }
  ]

Calibrate set thresholds from a baseline run:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_metric_set"
  action: "calibrate"
  resource_id: "<set-uuid>"
  body:
    run_id: "<baseline-run-uuid>"
    factor: 0.9                  # optional, default 0.9

Step 3: Models

Register a model. Required: name, provider, model_id (provider's model id, e.g. claude-sonnet-4-6). The Harness secret reference is api_key_secret_ref (not secret_ref):

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_model"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "Claude Sonnet 4.6"
    provider: "anthropic"        # openai | anthropic | google | azure | custom
    model_id: "claude-sonnet-4-6"
    api_key_secret_ref: "<harness-secret-id-for-api-key>"
    default_temperature: 0.2
    default_max_tokens: 4096
    is_active: true

Step 4: Targets

Register an invocation target. For storage_type='managed', both type and config are required. For git-backed, omit type/config and provide storage_type='git' + git_source:

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_target"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "Support Bot v3"
    type: "prompt"               # prompt | agent | precomputed
    config:
      prompt_ref: "<eval_registry_item-uuid>"
      model_id: "<eval_model-uuid>"
    is_active: true
    env_secrets:
      OPENAI_API_KEY: "openai_pat"
    connector_ref: "http_target_connector"

For precomputed targets, upload outputs after creation:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_target"
  action: "upload_outputs"
  resource_id: "<target-uuid>"
  body:
    items:
      - item_identifier: "row-001"
        output: { answer: "..." }
        metadata: { latency_ms: 120 }

Test a target with a sample input:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_target"
  action: "test"
  resource_id: "<target-uuid>"
  body:
    input: "What is our SLA?"

Step 5: Evaluations and runs

Create an evaluation. The eval starts in draft until dataset_id, target_id, and metric_set_id are set:

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "evaluation"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "Support Bot Quality"
    description: "Quality regressions for support-bot-v3"
    dataset_id: "<dataset-uuid>"
    target_id: "<target-uuid>"
    metric_set_id: "<set-uuid>"
    sampling_strategy: "all"     # all | random | first_n
    concurrency: 5
    cost_limit_usd: 25
    timeout_per_item_ms: 30000

Activate it via harness:harness_update with status: "active". Then run:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "evaluation"
  action: "run"
  resource_id: "<eval-uuid>"
  body:
    trigger_type: "manual"        # manual | scheduled | api | ci
    sample_size: 100              # optional override
    run_inputs:
      model_id: "<override-model-uuid>"

Compare two runs:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_run"
  action: "compare"
  body:
    run_ids: ["<run-a>", "<run-b>"]

Rescore an existing run with updated metrics:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_run"
  action: "rescore"
  resource_id: "<run-uuid>"
  body:
    metric_set_id: "<new-set-uuid>"

Drill into per-item results:

Call MCP tool: harness:harness_list
Parameters:
  resource_type: "eval_run_item"
  filters:
    run_id: "<run-uuid>"

Step 6: Suites

Group evals into a suite with a cron schedule. Important: pass_strategy values are all_must_pass or weighted_threshold (not "all" / "majority"). schedule is an object, not a string. Members are not added in the create call - use replace_evaluations afterwards:

Call MCP tool: harness:harness_create
Parameters:
  resource_type: "eval_suite"
  org_id: "<org>"
  project_id: "<project>"
  body:
    name: "Nightly Regressions"
    description: "Catches model drift on the support agent"
    purpose: "release_gate"          # pr_gate | cd_gate | release_gate | custom
    pass_strategy: "all_must_pass"   # or weighted_threshold (with pass_threshold)
    is_blocking: true
    schedule:
      cron: "0 2 * * *"
      timezone: "UTC"
      enabled: true

Add member evaluations:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_suite"
  action: "replace_evaluations"
  resource_id: "<suite-uuid>"
  body:
    entries:
      - { evaluation_id: "<eval-1>", is_required: true, position: 0 }
      - { evaluation_id: "<eval-2>", is_required: true, position: 1 }

Run the suite on demand:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_suite"
  action: "run"
  resource_id: "<suite-uuid>"
  body:
    trigger_type: "manual"

List suite runs:

Call MCP tool: harness:harness_list
Parameters:
  resource_type: "eval_suite_run"
  filters:
    suite_id: "<suite-uuid>"

Step 7: Analytics, annotations, registry, git

eval_analytics - singleton; harness:harness_get (no resource_id) returns the project summary.
eval_annotation - log human or automated trace annotations; histogram execute action returns label distribution.
eval_registry_item - register reusable prompts, agents, MCP tools, and skills (type: prompt | agent | mcp_tool | skill).
eval_git_settings - singleton; harness:harness_get / harness:harness_update to configure git sync.

Step 8: YAML import / export

For batch / IaC workflows, use import_yaml to create targets, datasets, metric sets, and evals (or full suites) in one shot. Always set dry_run: true first.

Validate an evaluation YAML (creates target + dataset + metric set + eval together):

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "evaluation"
  action: "import_yaml"
  body:
    yaml_content: "<full evaluation YAML>"
    dry_run: true
    auto_run: false

Validate a suite YAML (creates suite + member evaluations together):

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_suite"
  action: "import_yaml"
  body:
    yaml_content: "<full suite YAML>"
    dry_run: true

Export a suite (or eval) for version control:

Call MCP tool: harness:harness_execute
Parameters:
  resource_type: "eval_suite"
  action: "export_yaml"
  resource_id: "<suite-uuid>"

Examples

"Set up nightly regressions for the support bot" -> eval_dataset -> eval_target -> eval_metric_set (with entries) -> evaluation -> activate -> eval_suite with cron object schedule + replace_evaluations
"Compare last week's run to today's" -> harness:harness_execute eval_run compare with run_ids
"Rescore that run with the new metric set" -> harness:harness_execute eval_run rescore
"Generate 100 synthetic test questions" -> harness:harness_execute eval_dataset generate (strategy=use_case, count=100, model_id) then poll_generate
"Validate this eval YAML before applying" -> evaluation import_yaml with dry_run=true

Related skills

Prerequisites (skills typically run before this one):

/managing-secrets

Composes with (skills commonly invoked together):

/creating-agent-templates

Follow-ups (skills commonly chained after this one):

/reporting-dora-metrics

Performance Notes

Dataset generation is async - always poll poll_generate rather than blocking.
For large datasets, prefer bulk_upsert over many single-item creates.
Use eval_run_by_eval to filter runs by eval_id instead of paginating all runs.
import_yaml with dry_run=true is the cheapest validator - use it before every real apply.

Troubleshooting

"dataset_id required" - eval_dataset_item operations need params.dataset_id. Same pattern for eval_metric_set_entry (set_id), eval_run_item (run_id), eval_suite_evaluation (suite_id).
Run never finishes - check eval_target is reachable; if type: prompt, confirm the model's api_key_secret_ref resolves.
Calibrate fails with "no baseline" - eval_metric_set.calibrate requires a run_id body parameter pointing at a real prior run.
Suite create rejects schedule: "0 2 * * *" - schedule is an object, not a string. Wrap it as { cron: "0 2 * * *", timezone: "UTC", enabled: true }.
pass_strategy: "all" rejected - values are all_must_pass or weighted_threshold. Use pass_threshold (number 0-1) when strategy is weighted_threshold.
evaluations array on suite create has no effect - the suite create body has no top-level evaluations field. Use replace_evaluations after creating the suite.
prompt_ref / model_id on eval_target rejected - those go inside config, not at the top of the body. Body needs type + config for managed targets.
secret_ref on eval_model rejected - field is api_key_secret_ref.
Schema drift - run harness:harness_describe(resource_type='<eval_*>') for the live schema if any of the above fails.