agentsclimarketplace

Skill tuning loop

Skill patrickserrano/lacquer/core/skills/skill-tuning-loop

Go CLI + profile templates that standardize how Claude Code works across every project

Install
npx -y skills add patrickserrano/lacquer --skill skill-tuning-loop

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Empirically test whether a lacquer skill needs an edit by mining real session transcripts for friction, proposing a bounded fix, and validating it against held-out cases before it ships. Use when a skill in core/skills/ or profiles/*/skills/ keeps getting corrected in practice, when triaging a scheduled tuning-pass PR, or when skill-authoring-standard's manual rubric isn't enough to tell whether a proposed edit actually helps. Distinct from skill-authoring-standard (a static prose rubric checked by a human) — this is an empirical, evidence-gated loop checked by rollouts.

SKILL.md

5.1 KB, as published. Nobody here has run it

Skill Tuning Loop

A skill earning its place in the lacquer (per skill-authoring-standard) is a judgment call today. This loop adds a second, empirical bar: does a proposed edit measurably reduce the friction real sessions hit, without regressing what already works? Run it via the skill-tuning-loop Workflow — never edit a skill's behavior-affecting content based on mined evidence without going through the validation gate below.

When to run this

  • A skill keeps needing the same correction across unrelated sessions — that's a signal worth mining, not a one-off to shrug off.
  • Before accepting an automation-proposed edit (see the continuous-tuning design this skill's workflow implements) — the gate is what makes an unattended proposal safe to merge.
  • Never on every session, and never on a single session's evidence — one grumpy correction is noise, not a pattern. Require independent evidence from multiple sessions before proposing a change.

The loop (implemented in the skill-tuning-loop Workflow)

  1. Mine. Search recorded sessions (via the agentsview MCP tools) for invocations of the target skill plus friction phrases ("no that's wrong", "actually", "don't do that"). Pull the surrounding messages for anything that looks skill-relevant.
  2. Reflect. Have an agent read the current SKILL.md against that evidence and name concrete, recurring patterns — what the skill says (or omits) and what went wrong because of it. Require ≥2-3 independent sessions per pattern; thin evidence gets flagged, not acted on.
  3. Propose. Generate a bounded edit — add/replace the few lines implicated by the evidence, not a rewrite — that still satisfies skill-authoring-standard (trigger-oriented description, instruction over exposition, no padding).
  4. Validate. Run the same small held-out task set (references/eval-cases.md if the skill has one, else 3 representative tasks derived from its description) against the old and new skill text in parallel, judged by an independent agent running on a stronger model than the one that drafted the proposal — the same posture as advisor-checkpoint, spending the expensive model at the one load-bearing moment (the ACCEPT/REJECT verdict) rather than throughout. Accept only if the new version doesn't regress any case and strictly improves at least one. A good eval case is a task plus an explicit, checkable success criterion — not "handles it well," but the specific behavior that counts as a pass — and the set should include at least one edge or ambiguous case, not just the obvious path. If you're authoring references/eval-cases.md by hand for a skill, hold it to that same bar.
  5. Report, don't merge. A REJECT verdict ends the loop — log the rejected proposal and reasoning so a future pass doesn't re-propose it without new evidence. An ACCEPT verdict produces a diff and its evidence (session excerpts, before/after eval scores) for a PR — a human merges it. This loop never commits, pushes, or merges on its own.

Guardrails

  • Bounded edits only. A diff that touches more than the lines implicated by the evidence is a rewrite wearing a tuning-loop's clothes — reject it and re-propose narrower.
  • Validation gate is non-negotiable. Mined evidence proposes; held-out eval disposes. Skipping the gate turns this into "one user's bad session rewrites a shared skill," which is worse than not running the loop at all.
  • Human merges. Same posture as this repo's git-safety rules generally — an accepted proposal is a reviewable PR with its evidence attached, never an auto-applied change.
  • A "this tool/skill doesn't exist" finding is never trustworthy on its own. Reflect and Propose can only grep this repo's own committed files — they have no visibility into Claude Code's system-level tools (ScheduleWakeup, EnterPlanMode, CronCreate, TaskCreate, ...) or plugin-provided skills installed outside core/skills//profiles/*/skills/ (flowdeck, swift-concurrency-pro, ...). A real run proposed "fixes" for all three of those, replacing a correct reference with worse text because the tool/skill was real but invisible to a repo-only grep. The workflow script forbids finding, proposing, or accepting an edit based on such a claim; still spot-check any accepted proposal whose rationale leans on a tool or skill "not existing" before applying it — this class of mistake reads as confident and well-verified right up until it isn't.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.