agentsclimarketplace

Native agent skill validation

Skill Kbediako/evergreen-codex-skills/skills/native-agent-skill-validation

Evergreen, generally useful Codex skills designed to stay valuable as models improve.

Install
npx -y skills add Kbediako/evergreen-codex-skills --skill native-agent-skill-validation

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 21 days oldThe repository was created 21 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Forward-test an existing or changed Codex skill with fresh, contamination-resistant native-agent trials. Use to test whether instructions trigger, generalize, and change behavior under a fixed Sol worker condition; do not use to create or update the skill itself.

SKILL.md

3.8 KB, as published. Nobody here has run it

Native Agent Skill Validation

Establish Ownership

Use the runtime-provided skill-creator skill for skill creation, editing, package structure, and basic validation. Begin this workflow only after a candidate skill exists.

Load native-subagents-first before using native agents. Follow it as the sole lifecycle, model, budget, resume, and direct-tool contract.

Define The Forward Test

  1. State one behavior the skill should cause.
  2. Define pass, fail, and inconclusive criteria before testing.
  3. Build a realistic user-style task from raw artifacts.
  4. Withhold the intended answer, suspected defect, proposed fix, and prior conclusions.
  5. Keep permissions and side effects no broader than the real task requires.

Hold Conditions Fixed

Compare treatment and control with:

  • the same Sol model and reasoning effort;
  • the same role, tool surface, prompt, permissions, and artifact snapshot;
  • a fresh child and uncontaminated workspace state for every trial;
  • only the candidate skill condition changed.

Do not reuse children across treatment and control. Do not leave prior outputs, patches, filenames, or evaluator notes where later trials can discover them.

Mark any model, effort, prompt, tool, permission, or artifact mismatch as a confound. Do not attribute the outcome to skill text until rerun under matching conditions.

Run And Verify

  1. Run the basic quick_validate.py check from skill-creator.
  2. Choose the claim:
    • Natural trigger: leave candidate metadata available and use a realistic prompt that does not name the skill. Audit loaded, read, announced, and applied separately. This tests retrieval and usability, not revision effect.
    • Revision effect: compare isolated prior-revision and candidate-revision conditions under the same prompt; verify which body each condition read and applied.
  3. Treat explicit invocation as invoked-usability evidence, not natural-trigger or revision-effect evidence. Never label a run "without the candidate" when it could read or apply it.
  4. Persist exact prompt bytes or hashes, skill revisions and availability, run order, session/rollout IDs, and structural invocation evidence outside the test workspace before spawning.
  5. Wait for and consume each final child answer.
  6. Verify cited files, outputs, commands, and mutations in the parent.
  7. Treat status as lifecycle evidence only; never treat a completed status as proof the task succeeded.
  8. Repeat independent isolated comparisons before promoting a behavior claim.
  9. Send instruction defects back to skill-creator for editing, then start a fresh validation cycle.

Use at least two independent matched trials per condition for a behavior-promotion claim. Use a single matched pair only as exploratory evidence and label it non-promotable.

Judge

Check whether:

  • frontmatter triggered for the right request;
  • the body and directly linked references supplied enough guidance;
  • behavior improved under the fixed worker condition;
  • the result generalized beyond wording copied from the skill;
  • final evidence survived parent verification;
  • no contamination or unintended mutation occurred.

Report

  • Candidate skill and behavior claim.
  • Fixed Sol worker condition.
  • Control and treatment prompts.
  • Trial count and fresh-state method.
  • Exact skill availability/revisions, prompt identity, run order, and rollout IDs.
  • Child final evidence and parent verification.
  • Pass, fail, or inconclusive result.
  • Confounds and whether the result is exploratory or promotable.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.