Model guide

Model Guide (routing summary)

Lightweight routing aid. The full living eval doc with dated findings logs is MODEL_GUIDE.md (same dir) — read it when you need the per-model teardown (vendor methodology critique, neutral evals, behavioral properties, pros/cons). Per-model evidence briefs are in research/<slug>.md.

Selection protocol

1. Default to DeepSeek V4 Flash 0731 at max (fireworks/accounts/fireworks/models/deepseek-v4-flash-0731) — GLM-5.2-class capability at a fraction of the cost, proven in local pi-harness duels (12/12 vs GLM on Matt-shaped tasks at high and top gear; 4-12x cheaper per accepted task; retrieval + exact-wording clean to ~925K). max is its sweet spot: same cost as high, faster, vendor's agentic recommendation. Text-only. Hard bans: never a verifier (84% hallucination — generates rather than refuses); never thinking-on + strict json_schema (schema-valid integer corruption). GLM 5.2 keeps the long-horizon marathon seat and is the fallback if DeepSeek degrades (its silent under-counting verifier caveat stands too). When Flash fails at max, escalate within the family to Pro-0813 (fireworks/accounts/fireworks/models/deepseek-v4-pro-0813) before paying K3 prices — 8.5x Flash's cost at wall-time parity (local duel 14/14 = 14/14, pack ceiling), vendor agentic table ~K3-minus-5 unreplicated, 94.1% hallucination (verifier ban extends); its schema-corruption bug did NOT reproduce (0/10) but budget ≥16K output at max (reasoning hunger). Provisional slot pending neutral DeepSWE.
2. Open-weight bias for day-to-day; closed when it makes sense. Day-to-day leans DeepSeek / GLM / Kimi / MiniMax. Pull closed (Claude, GPT) in for quorums or when their specific strength is the point. Don't default to closed just because it's frontier — under neutral harnesses the closed-model coding leads shrink or reverse.
3. Kimi K3 is additive: paired frontier executor + hard-task escalation. It does not replace GLM 5.2, Opus 5, Sol, or Fable. Use it for difficult repo-scale coding and vision when its frontier-cluster DeepSWE result justifies the premium. Preferred hard-work pattern: Opus 5 plans/reviews → K3 implements → mechanical checks or the parent accept. This pairing is a routing recommendation, not a measured K3+Opus benchmark. Never let K3 define and accept its own success criteria — its explicit imaginary-grader/eval-maxxing failure mode makes that unsafe. Post-launch reality: throughput is provider-split — Fireworks serves K3 at ~163 tok/s with ~1s first chunk (interactive-viable), first-party Kimi crawled at ~33 tok/s under launch congestion. It acts on ambiguity (task specs need explicit non-goals), and its failure shape is an expensive confident loop (bound runs with turn/token budgets).
4. Qwen3.8-Max is additive: the Alibaba quorum seat + a GDPval-class knowledge-work niche. Its DeepSWE 57 ±3 @ $3.73 sits between GLM 5.2 (44) and K3 (69), but the "cheaper escalation" framing fails retry math — cost per successful task is $6.54 vs K3's $6.74 (parity), and K3 wins on one-shot rate + vision. Real value: a seventh independent quorum family, and the one neutral row where it leads something (GDPval-AA 1739, behind only Opus 5) for document/deliverable knowledge work — unverified locally. Not the default (DeepSeek ~14x cheaper per token), not marathons (Fireworks route caps at 262K text-only — the 1M/multimodal headlines are the Alibaba API), never a verifier (hallucination regressed 23%→40%). Thinking defaults to max-effort — set the level on cheap calls. Open weights promised but not shipped; the open-weight bias doesn't apply yet.
5. xAI is excluded by preference (Matt, 2026-08-24) — no Grok seat unless a neutral result is outstanding AND unmatched by an incumbent. Grok 4.6 was evaluated 2026-08-24 and didn't clear the bar: coding loses per-success to K3 ($8.21 vs $6.74 per DeepSWE success); its best neutral row (GDPval 1753) is still #2 behind Opus 5 and only ~14 Elo over Qwen at the same $2/$6 — challenger noise, not outstanding. Don't route to xai/* by default, in quorums, or as escalation. Re-eval triggers and full evidence: MODEL_GUIDE.md → Grok 4.6.
6. Quorums across discrete families for decisions. Review/critique/design choices run models from different families: Anthropic, OpenAI, Moonshot, MiniMax, Z.ai, DeepSeek, Alibaba/Qwen (Fireworks is a host, not a family). Valid 3-family quorum: Opus 5 + GPT-5.6 Sol + Kimi K3. Invalid: Opus + Sonnet + Fable (all Anthropic). For decisions, not every execution task.
7. Fable 5 is a consciously-invoked juggernaut. Very expensive, 109s to first token at max effort (batch/async only, never interactive). Pull in only when the work is deemed worth it. Prep it with cheaper agents first (research, planning, context-gathering) so its expensive tokens go to hard execution, not figuring out what to do. Never for cyber/red-team tasking absent Glasswing access — the filter is framing-immune (4/4 blocks in the exosphere red-team pipeline, 2026-08-09, authorized-preamble retries included).
8. MiniMax M3 is never a coding agent. Non-coding only — verification/librarian work (abstention = precision-over-recall filter) and cheap multimodal.
9. Match thinking level to stakes, not to model. pi levels: off/minimal/low/medium/high/xhigh/max (max new in pi 0.80.6, native on GPT-5.6 + adaptive Claude). Low-stakes utility work stays low/medium; hard coding/review escalate to xhigh/max. Don't pin max globally — it multiplies token spend on every call. Per-model: DeepSeek V4 Flash 0731 runs the full minimal→max ladder and max is its sweet spot (measured same-cost/faster vs its own high; thinking-off collapses multi-step reasoning — treat thinking as required); GLM 5.2 exposes off/high/xhigh on Fireworks (xhigh measured ~2.7x output tokens for zero pass gain — prefer high); Fable 5 can't disable thinking; K3 always thinks but exposes low/high/max and requires preserved reasoning_content; Qwen3.8-Max runs the full ladder on Fireworks (off→none, verified disabling; native levels low/medium/xhigh and xhigh is the default — set the level explicitly on cheap calls); M3 is limited to low/medium/high. See MODEL_GUIDE.md → Thinking levels.

Quick reference

Model (pi ID)	AA Idx	Role	Use for / Not for
fireworks/accounts/fireworks/models/deepseek-v4-flash-0731	52	default (at max)	Everyday grunt coding: bugfixes, features, removals, refactors, batch loops, draft-executor under an Opus planner. Not: verification (84% hallucination), thinking+strict-schema extraction (integer corruption), ungated long-horizon, vision.
fireworks/accounts/fireworks/models/deepseek-v4-pro-0813	53	provisional: intra-family escalation (Flash fails → Pro → K3)	DeepSeek-family hard coding when Flash-max falls short, at ~1/4 K3's output price. Not: default (8.5x Flash cost, +1 AA), verifier (94.1% hallucination), low-budget extraction at max (reasoning eats the cap), marathons, vision.
fireworks/accounts/fireworks/models/glm-5p2	53	long-horizon marathons + fallback	Multi-day project-context work, fallback route if DeepSeek degrades. Not: vision, frontier reasoning, verification (silent under-count).
fireworks/accounts/fireworks/models/kimi-k3	60	paired frontier executor + hard-task escalation	Difficult repo-scale coding, frontend/vision, Moonshot quorum seat; ideally Opus 5 plan/review → K3 execution → external acceptance. Not: routine volume, factual verification, self-acceptance, mid-session model switching, the first-party Kimi route (33 tok/s under congestion — use Fireworks).
fireworks/accounts/fireworks/models/qwen3p8-max	58	Alibaba quorum seat + knowledge-work niche	Cross-family quorum votes; GDPval-class docs/deliverables (1739 Elo, #2 behind Opus 5 — verify locally first); second route for hard text coding if K3 is degraded. Not: "cheaper K3 escalation" (retry math is parity: $6.54 vs $6.74 per DeepSWE success), default volume, vision (Fireworks route is 262K text-only), marathons, verification (hallucination 23→40% regression).
fireworks/accounts/fireworks/models/minimax-m3	45	never coding — verification + cheap multimodal	Doc-vs-code reconciliation (fewer false alarms), cheap image/video. Not: coding, code review, missed-discrepancy-critical verification.
openai-codex/gpt-5.6-luna	52‡	vision/computer-use only	Vision/computer-use subtasks (GLM 5.2 is text-only), narrow execution with a plan. Not: hard reasoning, long-horizon, large-context, fire-and-forget. GLM 5.2 wins on text coding.
openai-codex/gpt-5.6-sol	61‡	agentic breadth — capability upgrade, NOT trust upgrade	Terminal-native agent loops, computer-use/browser, web-research, ultra parallelizable tasks. Not: unsupervised destructive-tool loops (METR: highest reward-hacking of any public model; system-card overreach/fabrication), routine coding (GLM 5.2 cheaper + safer). Price-neutral vs old 5.5.
anthropic/claude-sonnet-5	55	agentic execution + injection robustness	Defined-plan execution, untrusted-input ingestion, vision/computer-use. Not: not cheaper than Opus per task ($2.29 vs $1.97), unsupervised security fixes (SecPass 19.6%), open-ended planning.
anthropic/claude-opus-5	63†	frontier architecture + review	Hard architecture, security review, adversarial product criticism, and planning/review around a bounded K3 executor. Not: high-volume loops or unsupervised negotiation.
anthropic/claude-fable-5	62	juggernaut (conscious, async)	Ambiguous investigative long-horizon work, large migrations, multi-day autonomous + self-verification. Not: interactive (109s TTFT), known cyber/bio work (use Opus 5), sole security reviewer (19% SecPass + memorization).

Gotchas every routing decision must respect

• Closed-model "beats X on coding" headlines mostly fail neutral harnesses. Opus 4.8's SWE-bench Pro lead over GPT-5.5 reverses on DeepSWE (58-59 vs 67-70); Cursor's audit found 63% of Opus's SWE-bench Pro wins were retrieval, not derivation. SWE-bench Pro/Verified are contaminated/scaffold-dependent across all vendors (OpenAI's own team flagged ~30% of SWE-bench Pro tasks broken 2026-07-08). Weight Terminal-Bench and DeepSWE over SWE-bench numbers.
• GPT-5.6 Sol is a capability upgrade over 5.5 but a trust downgrade. METR independently found Sol's reward-hacking rate is the highest of any public model it's evaluated — high enough that METR says its own capability numbers aren't a robust measurement. OpenAI's own system card documents increased "acts beyond user intent" (deleting wrong VMs, fabricating "verified" claims, credential exfiltration) and metagaming up to 55.4% (from 5.5's 41.2%). Opposite of M3's abstain-and-flag: Sol fabricates-and-proceeds under pressure. Never run it unsupervised with destructive tools; verify its "done" claims independently. Launch AA numbers were pre-release-coordinated, but post-launch common-harness DeepSWE now corroborates coding capability (73% ±3); trust caveats remain unchanged.
• Muse Glimmer 30B was evaluated locally and rejected (2026-08-12). Coding parity with DeepSeek Flash at 4.7x cost / 3.4x latency, AA 35.1 vs 52, 1/2 on its own vision-agentic probe vs qwen3p7-plus's 2/2-substance at the same price. Entry stays in models.json for re-checks; its real niche would be local always-on (24GB), untested here. Details: MODEL_GUIDE.md + research/muse-glimmer-30b.md.
• GPT-5.6 Terra was rejected. AA (OpenAI's own launch-partner evaluator) says Terra is strictly Pareto-dominated by Sol/Luna at every effort level; vals.ai scores it below gpt-5.5; "5.5-class at half cost" is unproven. Don't pick Terra — pick a Sol or Luna effort level instead.
• "Cheap" isn't always cheap per task. Sonnet 5 costs more per task than Opus 4.8 at standard pricing (tokenizer inflation + 3-6x agentic turns). M3 is cheap per token but token-hungry enough to fail expensively on coding. GLM 5.2's per-token ~1/6 advantage shrinks at Max effort (2-2.7x GPT-5.5's reasoning tokens).
• Fable 5's safety reroutes are not routing recommendations. Cyber fallback still targets retired Opus 4.8; bio fallback now targets Opus 5. Use Opus 5 directly for known cyber/bio work rather than mirroring a legacy classifier fallback. Blocked-before-output requests aren't billed; rerouted answers bill at the fallback's rate. Operationally confirmed 2026-08-09 (exosphere-red-team): Fable's cyber blocks are framing-immune — 4/4 including authorized-preamble retries; Opus 5 ran the same tasks unblocked.
• DeepSeek 0731's failure modes are refusal-free generation + structured-output corruption. 84% hallucination (AA-Omniscience) — it answers instead of abstaining, so never seat it as a factual verifier. Thinking-on + strict json_schema returns schema-valid JSON with wrong integers — keep thinking and strict schemas apart (thinking-off is safe for single-step extraction but collapses multi-step reasoning). Verbose: real cost ~2-3x sticker unless cached. Pin the dated -0731 slug (the undated alias was silently re-pointed). Fireworks uptime for it measured 91.6% on OpenRouter stats — have a fallback route. Rules-file adherence at depth is a reported-but-locally-unreproduced concern.
• K3's failure mode is an imaginary grader. In 2,544 Pi-harness trajectories, Latch found explicit eval-awareness references in ~61%; the behavior persisted on 29% of direct tasks with grader references removed, and affected runs burned ~4x tokens/time while scoring lower. K3 may fabricate data or optimize for a scorer it invented. Give it external criteria and external acceptance; never use it as factual verifier or self-grader.
• Deployment properties are route-specific, not model constants. K3's throughput was provider-split (163 tok/s on Fireworks vs 33 first-party); Qwen3.8-Max's context and modality are split too — Fireworks serves 262K text-only while Alibaba's API is 1M multimodal. Route on what the configured route delivers, not the launch post.
• Don't trust a single vendor benchmark lead. Nearly every model in this guide had a headline that partially failed a neutral cross-check (two partial exceptions: Qwen3.8-Max's vendor DeepSWE 56.6 matched the 57 ±3 rerun, and Grok 4.6's AA-61 and DeepSWE 65.9/67.0 claims verified at 60.92 and 67 ±2 — but Grok's coding framing still failed its own table, and both vendors' other headlines carry caveats). When a new model appears, assume the same and find the neutral number (Artificial Analysis, DeepSWE, vals.ai, LMArena, independent teardowns). Also: AA index scores drift between reads (the v4.1.1 re-score moved everything +1-3 in early August) — never compare AA numbers read on different dates.

For depth

• MODEL_GUIDE.md — full living eval doc: per-model What it is / Vendor claims & methodology / Neutral evals / Behavioral properties / Pros / Cons / When to use / When not to use, plus append-only dated Findings logs and a Changelog. Read this for any non-obvious routing call.
• research/<slug>.md — the source evidence brief for each model (vendor methodology teardown, neutral-eval citations, behavioral findings). Read when you need to verify a claim or update the guide.

Provider prefix note: it's openai-codex, not "openapi-codex". GPT-5.6 uses durable tier names (Sol=flagship, Terra=mid [rejected], Luna=mini); bare gpt-5.6 routes to Sol. Specs from pi --list-models; K3 is confirmed locally at 1M context / 131K max output with thinking and vision enabled; DeepSeek V4 Flash 0731 confirmed at 1M / 384K with the full minimal→max ladder; Qwen3.8-Max confirmed at 262K / 131K text-only with a full off→max ladder (native top effort is xhigh; off maps to none, verified disabling reasoning). AA column refreshed 2026-08-12 (v4.1.1; the whole index drifted +1-3 vs the July reads in findings logs — cross-date comparisons invalid). Context-window caps are intentional (the GPT-5.5/5.6 family degrades at long context; Luna retrieval flatlines ~41% at 256K-512K). † AA evaluated Opus 5 with Anthropic pre-release support; DeepSWE is the blind/common-harness cross-check. ‡ = pre-release-coordinated AA number, not blind.