Model profiler
Impartially PROFILE the cross-provider sub-agent fleet against the FIXED 14 work-categories (directly benchmarked parents + 4 composite-inferred) whenever a new model ships (or on demand). Discover every model published in the recent window by the in-scope provider families, gather ALL public benchmark scores + statistics, map them onto the directly benchmarked parent categories (composites composed from parents, never directly benchmarked), then JUDGE each model+effort pairing into per-category tier rankings (best→worst) SOLELY from the discovered research, with a recorded rationale per tier placement. Emits exactly 3 artifacts: routing-table.json (lean), routing-table-audit.json (full provenance), research-seed-sites.json (accumulating learned source list). Single-family and multi-family are both fully-supported, first-class paths; provider mix is optional. The 14 categories are immutable inputs : this skill never derives, chooses, renames, reorders, or reshuffles them. Use when a new model is released, when asked to profile new model, re-profile models, re-profile the fleet, rebalance routing, update routing table, refresh model profiles, re-run model research, regenerate routing-table.json, refresh tier rankings, or to answer "which model for X now" after a model launch. Orchestrator-only pipeline: model discovery + maximalist benchmark research, pivotal-question interview, flagship judging + merge, 3-artifact emission, 3-pass adversarial validation, and provider/seed validators + scenario routing tests. Sub-agents dispatched via `mcp__subagent-mcp__launch_agent`; cross-family critics are available when ≥2 families are reachable.From its SKILL.md
npx -y skills add Heretyc/subagent-mcp --skill model-profilerAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
16.2 KB, as published. Nobody here has run it
Model Profiler
Impartially profile the sub-agent fleet against the FIXED 14 work-categories (directly benchmarked parents + 4 composite-inferred) when a new model ships (or on demand). The skill is the impartial judge of all models: it discovers the models, gathers their public benchmarks, and ranks each model+effort pairing per category : it does not decide what the categories are.
Input = the profiling scope (in-scope provider families + recent window) confirmed in Phase 0
or supplied by the standing repository profile when its exact trigger matches.
Output = EXACTLY 3 persisted artifacts (src/routing-table.json, src/routing-table-audit.json,
research-seed-sites.json); nothing else persists. See the Output Contract below.
This SKILL.md is the index : load only the current phase's detail leaf, never all of them. Each md file stays <=200 lines (AGENTS.md cap).
Required Runner (read first)
Run ONLY on the highest available flagship model the operating provider offers (whatever
that currently is), at its highest OR second-highest effort setting (i.e. a top-tier
reasoning model at high effort; the provider-equivalent top model+effort otherwise). Note:
binding an explicit runner model/effort for sub-agents is itself gated : see the gating
preamble in references/dispatch-mechanics.md (smart mode rejects selector-bearing launches
unless the user-approved-overrides window is open). It is
orchestrator-only: the runner dispatches every research/judging/validation step via
mcp__subagent-mcp__launch_agent and must sustain multi-phase reasoning across the whole
run. Never run on Haiku, a non-flagship tier, an effort below second-highest, or any
model lacking sub-agent-launch support or long-horizon reasoning : these silently
degrade the pipeline. If the runner does not meet this bar, halt and escalate to the
owner; do not run it. (Runner requirement only : distinct from invariant #2's ban on the
skill naming a preferred model for the JUDGED routing output.)
Fixed Taxonomy (immutable input : never derived here)
The 14 categories (directly benchmarked parents + 4 composite-inferred) + fallback_default@99
are fixed and immutable. Precedence order:
math_proof > security_review > debugging > quality_review > architecture >
agentic_execution > data_analysis > coding > knowledge_synthesis > mechanical >
prompt_engineering > vulnerability_research > molecular_biology > ml_accelerator_design
Tiles 11:14 are composite-inferred: no benchmark alias, never directly benchmarked : competency
composed from their parent tiles (docs/spec/task-taxonomy/composite-inferred-tiles.md).
fallback_default @99 : off-spine no-match catch-all; never one of the 14. Operational definitions
live in .spec/references/work-categories.md; determination methodology + rationale (incl. debate
provenance) live in docs/spec/task-taxonomy/. This skill profiles models against this spine.
Quick Decision Tree
| You are about to... | Load |
|---|---|
| Set up the worktree/branch gate + run the full git lifecycle (setup → commit → push → PR → deliver) : DO THIS FIRST | references/execution-lifecycle.md |
| Understand goal, I/O, hard invariants, orchestrator contract | references/overview.md |
| Run Phase 0 consent gate or apply the standing repository profile | references/phase-0-consent.md |
| Dispatch Phase 1 model-discovery + benchmark research / Phase 1.5 interview | references/phase-1-research.md |
| Check the canonical benchmark source list FIRST (run-to-run stability) | references/benchmark-sources.md |
| Run Phase 2 judging/arbitration + canonical merge | references/phase-2-synthesis.md |
| Emit the 3 artifacts (run builder + seed merge) | references/decompose-update.md |
| Run the 3-pass adversarial loop | references/adversarial-loop.md |
| Validate (validator + checklist + scenario routes) | references/validation.md |
Write a sub-agent prompt (dispatch via mcp__subagent-mcp__launch_agent) | references/dispatch-mechanics.md |
| Cite sources, apply labels, dogfood route tiers | references/citations-labels.md |
| Find the artifact being updated / provenance | references/artifact-map.md |
| Confirm the FIXED 14-category taxonomy + where its methodology lives | references/category-derivation.md (pointer) → .spec/references/work-categories.md |
| Understand tier ranking, interpolation, scoring formula, calibration gate | references/tier-ranking-and-scoring.md |
| Understand routing-table.json schema contract + validation rules | references/provider-json-emission.md |
Orchestration Entry Point: Phase 0 Detection
Exact prompt (whitespace-trimmed): Run the model-profiler skill. → apply the standing repository profile without consent prompts (authorized bare default run). Pre-conditions: CWD=repo root; prompt exact; no credentials/deletes/git-writes/taxonomy-changes/outbound-messages/out-of-allowlist requests.
When matched: apply the standing-profile answers (current-generation fleet, all reachable families, Fast mode, 90 min + session budget, provider mix optional), then follow the bare-run dispatch sequence (fresh Phase 1 every run; pre-existing scratch or budget/time shortfall → ABORT, never bounded-continuation, per the FRESH-DATA mandate → Phase 1.5→Phase 2 budget gate) in references/phase-0-consent.md. Non-matching prompts: load references/phase-0-consent.md and run the full AskUserQuestion consent flow.
(All of the above runs INSIDE the execution lifecycle : invariant #15: the worktree gate fires before Phase 0 detection, and the commit→push→PR→deliver lifecycle follows VALIDATE.)
Highest-Priority Mandates (OVERRIDE every numbered invariant below on any conflict)
- FRESH-DATA MANDATE (highest priority, non-negotiable). EVERY run MUST rank on FRESH research gathered THIS run; the prior audit (
src/routing-table-audit.json) and any prior committed rankings MUST NOT feed ranking/scoring : only SEED research (URLs / citations to re-investigate). If fresh data cannot be obtained and independently re-ranked this run : for ANY reason (budget/time, partial or pre-existing scratch, missing sources, interruption) : the run MUST ABORT: no continue, no bounded-continuation, no GAP-stub bypass for ranking; never degrade or resume from stale/prior data. (Reinforces #2: prior rankings diffed/flagged, never inherited.) - DELIVERY MANDATE. Every run ALWAYS OFFERS the full delivery lifecycle, never silently skipped: commit the 3 artifacts → push → open PR → resolve PR merge conflicts → mark PR ready → surface the clickable hyperlink for the user to MERGE the PR. (Extends #15's DELIVER box.)
Hard Invariants (Always Active)
- Fixed taxonomy. The 14 categories (directly benchmarked parents + 4 composite-inferred) +
fallback_default@99 are immutable inputs, defined in.spec/references/work-categories.md. This skill profiles models against them. It never derives, chooses, renames, reorders, merges, or reshuffles categories. A run that "discovers a missing category" is out of scope : surface it to the owner; do not act on it. - Impartial judging. Per-category tier rankings are derived solely from the Phase-1
discovered research, with a recorded one-line rationale per tier placement. No pre-baked rankings;
the prior KB is read only to diff/flag changes, never as the source of truth. The skill's
directives name no preferred provider/model/effort : only impartial role descriptors. The
judged model+effort rankings live exclusively in the OUTPUT artifacts (
routing-table.json/routing-table-audit.json), which are the profiler's product, not a directive. - Orchestrator-only. Once you have read the baseline workspace files (
AGENTS.md,work-categories.md, priorsrc/routing-table.json, priorresearch-seed-sites.json), you delegate all research, judging, validation, and writing to sub-agents. You dispatch, relay, persist, and decide : never execute the work yourself. - Sub-agents validate sub-agents. A producing agent never reviews its own output. Critics are FRESH agents distinct from producers (self-review ban / Anti-Pattern D), regardless of provider family. Cross-family critics are available when ≥2 families are reachable; on a single-family run, critics are fresh within-family agents : either way, never the producing agent itself.
- Provider mix is optional. Single-family and multi-family are BOTH fully-supported, first-class paths : neither is a degrade, neither is logged as risk; the profiler never requires a provider blend. A single-family run (e.g. Claude-only web-research) is equally valid. The only invariant is the fresh-critic / self-review ban (#4): critics must be distinct agents from producers in every case.
- Hub-and-spoke only. Sub-agents never call sub-agents. All coordination goes through you.
Inter-agent handoff is via
%TEMP%scratch files: full content to disk, only compact JSON status returned to the orchestrator. - Every sub-agent prompt begins with
<this is a request from a parent process>and the agent returns JSON{status, summary, source_locators, risks, writes_requested}. - Consent before dispatch. Phase 0 is a hard gate (
references/phase-0-consent.md). No sub-agent is launched before the owner confirms scope, mode, runtime/budget, and provider mix, or the exact standing repository profile trigger applies. Phase 0 confirms scope only : it must not preselect concrete models or efforts, and it never relaxes credential, destructive-action, outbound-message, git-write, or out-of-scope file protections. - Provenance purity. APA citations point to ORIGINAL external sources only. Never cite an
internal
.spec/references/*.mdfile as provenance. Label[SEED]/[INFERRED]/[ASSUMPTION]/[UNVERIFIED]. Seereferences/citations-labels.md. - Line caps. Every KB leaf and every skill markdown file stays <=200 lines.
- One spine, atomic update. The spine is still the one FIXED category spine; a run atomically emits the 3 routing artifacts together; the spine itself never changes, and a half-emitted state never reaches the default branch.
- Owner directive (absolute). Models with selectable effort settings must never emit
noneor other no-effort sentinels in the routing table, regardless of Phase 1 notes, vendor documentation, or other authority. Phase 1 agents exclude such pairings; Phase 2 judges silently drop them if discovered in research; the validator rejects them. This is enforced independent of the authority chain and overrides all cross-family research consensus. - DATA-ONLY boundary. This skill produces the 3 artifacts (
src/routing-table.json+src/routing-table-audit.json+research-seed-sites.json). It MUST NOT modifysrc/index.tsrouting logic. The only code it may touch:package.json,scripts/copy-provider.mjs,scripts/validate_provider.mjs,scripts/build_routing_table.mjs,scripts/update_seed_sites.mjs, andscripts/validate_seed_sites.mjs. - No-effort exclusion (cost_efficiency branch). Models whose ONLY effort is a no-effort
no-effort sentinel (
null/none/n/a: e.g.claude-haiku-4-5) are EXCLUDED from six parent categoriesagentic_execution,architecture,security_review,debugging,quality_review,knowledge_synthesis, but REMAIN ranked in the full-universe parentsmath_proof,data_analysis,coding,mechanical. Composite categories inherit parent eligibility: include a pairing if at least one parent includes it; compute simple mean over eligible parent ranks only. This is why no-effort exclusions are inherited automatically.build_routing_table.mjsenforces this at ranking;validate_provider.mjschecks coverage. Distinct from #12 (which bans emittingnonefor EFFORT-CAPABLE models): #14 EXCLUDES genuinely no-effort models from 6 parent categories. The authoritative effort ladder per model is the dataset'smodel_effort_universe; seeaudit.metadata.model_effort_universeas the single source of truth. On theperformancebranch, invariant #16's effort floor subsumes this rule (no-effort = belowhigh). - Execution lifecycle + worktree gate (ABSOLUTE : precedes every other step). Every
run executes inside ONE fixed, never-reordered, never-skipped lifecycle:
worktree/branch gate → main skill execution → commit → push → PR → resolve merge conflicts → PR ready → deliver PR link + change-summary. The FIRST action of any run : before Phase 0, before any read-for-write, before any dispatch : is the worktree gate:node scripts/check_worktree.mjsMUST printWORKTREE-GATE: PASS. ALL mutating work is FORBIDDEN in the primary tree; it happens only inside a compliant LINKED worktree on a<type>/<subject>branch OUTSIDE the repo. NativeEnterWorktreeis NON-COMPLIANT : use the manualgit worktree add. The final deliverable is the PR hyperlink plus a concise summary of what changed insrc/routing-table.jsonsince its last merged update. Full steps:references/execution-lifecycle.md; mandate:docs/spec/dev-loop/worktree-enforcement/. - Effort floors (owner directives 2026-06-11 + 2026-06-15 : FINAL AND BINDING).
Low effort is never ranked in any branch: the builder purges
lowfrom the global universe, validators reject it in committed artifacts, and callers cannot select it as a launch effort. Separately, theperformancebranch must NEVER rank any pairing belowhigh(null/none/min/light/medium) because performance/deadlock routing must not choose weak-effort variants.cost_efficiencymay still rank valid non-low efforts belowhigh.build_routing_table.mjs,validate_provider.mjs, andtest/performance-tier-effort.test.mjsenforce both rules on every build and commit. - Fresh-data (Highest-Priority Mandate : overrides #1:16). Rank on THIS run's fresh research; prior audit/rankings SEED only. Cannot gather & re-rank fresh data this run → ABORT (
blocked); no bounded-continuation, no GAP-stub bypass for ranking. - Delivery (Highest-Priority Mandate). Every run ALWAYS OFFERS commit → push → PR → resolve conflicts → PR ready → clickable MERGE hyperlink; never silently skipped.
Pipeline, output contract, and cross-links: references/pipeline-and-output.md.