Inference spec decode service

inference-spec-decode-service

Purpose

One orchestrator for "give a served model a draft tailored to its OWN traffic, and keep the standing config unless the tailored draft measurably wins." It does not re-implement any phase -- it sequences the mature per-phase skills behind a single, gate-driven, profile-matched loop and one evidence bundle, so an operator drives the whole profile -> corpus -> train -> A/B -> promote -> publish loop without hand-stitching skills together.

It is a self-hosted analog of Fireworks FireOptimizer's "adaptive speculative execution", and it differs operationally rather than algorithmically: the draft is trained on a corpus matched to the measured workload profile (the documented source of the higher draft hit-rate), and it is promoted only on a measured acceptance + TPOT win.

This is the closed-loop framing of inference-model-optimize Phase 6/7: that orchestrator trains a draft on a generic corpus one-shot. This one matches the corpus to traffic and is structured to run repeatedly.

Implementation (v1 oneshot). The loop is driven by specdec-loop.sh, which EXECUTES C3-C5 behind --i-understand-this-uses-the-cluster (DRY by default), with companions specdec-decide.py (the variance-controlled GO/NO-GO gate), canary-arm.yaml (the parameterized experiment-prefixed A/B arm), and specdec-presets.sh. The Workflow phases below map 1:1 to its C1..C5 + publish. --mode controller (drift-triggered re-adaptation) remains the v2 seam - see "Mode: controller" below.

When to use

• A served model has a latency ceiling you want to beat with a draft tailored to its actual traffic (not a generic UltraChat+ShareGPT draft).
• You want the "spec-dec as a service" capability: profile -> matched draft -> measured promote, as one workflow.
• You are standing up the loop now and want it architected to later become a deployed controller (continuous re-adaptation to traffic drift).

Do not use this skill for:

• A one-shot draft on a generic corpus with no workload profiling -- use inference-spec-decode-train directly.
• The full new-model bring-up (deploy/profile/tune/quantize/...) -- that is inference-model-optimize. This skill is its spec-dec sub-loop, made adaptive + repeatable.
• deepseek_mtp / ngram / Predicted-Outputs methods -- there is nothing to train (built-in head / prompt-driven / per-request reference text). Set the config and run only the acceptance A/B (Phase 4 below).

Example prompts

• "Stand up the spec-dec service for GLM-5.1 matched to our AA-shape traffic."
• "Auto-train a draft for this model's workload and promote it only if it wins."
• "Close the speculative-decoding loop: profile, match, train, A/B, publish."
• /inference-spec-decode-service --target /models/base --served <deploy> --profile access.jsonl --mode oneshot

Prerequisites

The skill fails closed if any of these are not satisfied.

1. A representative traffic sample (OpenAI-style access JSONL) OR a named bench-shape set (e.g. the inference-aa-workload shapes) to use as the profile.
2. The target base checkpoint (BF16/FP8 -- NOT the NVFP4 serving copy) staged on a shared FS/PVC the Slurm trainer can read (the inference-spec-decode-train prereq).
3. The standing served deploy + its current --speculative-config (the A/B baseline).
4. Serving-A/B cluster access. The canary runs on EITHER plain K8s (schedulerName: default-scheduler + nvidia.com/gpu) OR Slurm-on-K8s (the Slurm schedulerName + the slurm.example.com/lock toleration). The loop AUTO-DETECTS which (Slurm-on-K8s = a Slurm control plane + the lock taint) and --scheduler default|slurm overrides. For a TRAINED method (eagle3/dflash), also sbatch access to the SpecForge Slurm trainer (C3 is Slurm-native in BOTH cases). An idle GPU node confirmed BEFORE any GPU phase (Slurm-on-K8s: pin-node.sh-verified). Training never preempts serving. (The no-train deepseek_mtp/ngram path needs only the serving cluster.)
5. For a --mode controller run only: a writable service-state.json location.

Interaction style

Iterative + gate-driven, modeled on inference-model-optimize: run one phase, report the gate verdict + the headline number (DRAFT unless promoted), then ask before advancing. Never auto-advance past a red gate. The operator may start at any phase (--start-phase) or skip the train phases for a non-trained method.

Workflow

Phase 0: resolve intent + scaffold + service-state

State back the parameters: target-model-path, served-deploy, standing-config, profile-source (traffic JSONL | AA shapes | provided log), spec-method (eagle3 | dflash | deepseek_mtp | ngram | predicted_outputs), mode (oneshot | controller). Scaffold one evidence bundle (run-id == experiment-id == campaign) and initialize service-state.json:

{"schema":"specdec_service_v1","experiment_id":"<run-id>","target":"<path>",
 "served_deploy":"<name>","standing_config":"<json>","champion_draft":null,
 "last_profile_sha256":null,"last_ab_verdict":null,"history":[]}

service-state.json is the graduation seam to a deployed controller: it records the last profile hash, current champion draft, and last A/B verdict so a future controller Deployment can decide "has the workload drifted enough to retrain?" without re-deriving state. Report the scaffolded paths. Ask before any GPU phase.

Phase 1: profile the workload (WS-C1)

Hand off to inference-workload-profile: emit workload-profile.json (length distributions, content-class mix, ISL/OSL bench shapes, recommended_spec_decode.method). For an AA-shape profile source, use the AA-shape adapter (workload-profile.py --aa-shapes ...) so the three AA shapes (1k/10k/100k) become the profile. Record last_profile_sha256 in service-state. Gate: a profile with a method recommendation. If the recommendation is predicted_outputs (rewrite-heavy traffic), skip to Phase 4 with the per-request prediction path -- there is no draft to train.

Phase 2: build the hit-rate-matched corpus (WS-C2)

Run profile-to-corpus.py (in the inference-workload-profile skill dir): mode (b) converts a redacted traffic JSONL directly to the SpecForge conversations schema, mode (a) emits a weighted prepare_data.py plan matched to the content-class mix + length distribution. The corpus is the DATA_PATH for the trainer. Gate: a corpus that renders cleanly through the target chat template (a bad loss-mask shows up as training loss stuck at zero).

Phase 3: train the matched draft (WS-C3)

Hand off to inference-spec-decode-train with the Phase-2 corpus as DATA_PATH. The capture-correctness probe (--probe-logits) MUST pass before any full re-capture/retrain -- a broken hidden-state capture trains a head that serves at ~0% acceptance and wastes days of GPU time. Submit-and-queue on idle Slurm nodes. Never babysit, never preempt serving. Gate: non-degenerate training acc_0 / loss.

Phase 4: acceptance + TPOT A/B (WS-C4, the gate that matters)

Serve the trained draft via --speculative-config on an experiment-prefixed canary and measure in-engine acceptance (dflash_vllm_eval.py + vLLM spec_decode_* counters) on a same-node A/B vs the standing config, under cudagraph, >=3 trials (DRAFT->VERDICT). Gate: an acceptance-length win AND a median-TPOT win. A draft that loses is do-not-ship. Record last_ab_verdict.

Phase 5: promote on a win (WS-C5) + publish

On a measured win, flip the experiment-prefixed canary's --speculative-config (specdec-presets.sh renders the line). Set champion_draft in service-state. On a loss, keep the standing config and report why. Either way, publish the campaign: perf_tune_report_campaign_init --experiment-id <run-id> --focus latency -> cell_run (at the profile's bench_shapes) -> atlas_aggregate -> dcgm_correlate (raises sol_rigor to L3) -> report_render -> publish_to_lake. Record campaign=<id> + s3://perf-lake/... in SOURCE.md/summary.md.

Mode: controller (graduation seam -- NOT v1)

--mode controller is a documented stub for the deployed-service evolution: a long-running loop that periodically re-profiles live traffic, compares last_profile_sha256 to detect drift, retrains + re-A/Bs on drift, and auto-promotes on a win -- plus the dynamic runtime fallback (disable spec-dec when the live spec_decode_* acceptance drops). v1 implements only oneshot, controller reuses Phases 1-5 verbatim driven by a controller Deployment reading/writing service-state.json. Implementing the Deployment + fallback engine policy is the separate "(b) deployed controller" track.

Verdict rigor (DRAFT vs VERDICT)

Training acc_0 is a training-time signal, not a serve verdict. The promote/no-promote claim is the Phase-4 serve acceptance + TPOT A/B: DRAFT unless same-node + >=3 trials + metric-isolated (median TPOT/ITL) + matching cudagraph_mode + a production-representative baseline (the standing config, e.g. MTP K=3 -- never an eager strawman). The Phase-5 campaign publishes with the honest verdict_tier.

Safety

• Scheduling (plain K8s OR Slurm-on-K8s) -- the serving canary is rendered for whichever the loop detects (override with --scheduler default|slurm): plain K8s uses default-scheduler + nvidia.com/gpu. Slurm-on-K8s uses the Slurm schedulerName + the slurm.example.com/lock toleration, same-node-pinned to a pin-node.sh-verified-idle node (a bare nodeSelector to a non-idle node loops on NodeAffinity under the Slurm scheduler). TRAINED-method Slurm jobs (eagle3/dflash) submit-and-queue and never preempt serving. Preflight an idle node first.
• Standing config stays until a measured win -- the served --speculative-config is not changed until the Phase-4 A/B shows a real acceptance + TPOT win under cudagraph.
• Capture-correctness gate -- never launch a full re-capture/retrain until the --probe-logits probe passes. A broken capture costs days of wasted training.
• Experiment isolation -- canary + draft-staging objects experiment-prefixed + experiment=<id-slug>. Teardown by label, PV last. Never touch standing deploy names.
• Local/fork only -- no upstream PR / external posting without explicit per-turn operator approval.

Source-of-truth references

• v1 loop components: specdec-loop.sh (the loop), specdec-decide.py (GO/NO-GO gate), canary-arm.yaml (A/B arm), specdec-presets.sh (config rendering).
• inference-workload-profile -- Phase 1/2 (profile + corpus).
• inference-spec-decode-train -- Phase 3 (SpecForge train). Its capture-correctness + serve-wiring gates are the ones this loop encodes.
• inference-model-optimize -- the parent orchestrator whose Phase 6/7 this loop makes adaptive + repeatable.
• inference-aa-workload -- the AA-shape profile source.

Contact

Open an issue in this repository.

Full-context reporting (no bare numbers)

Per the methodology canon "Every performance number carries its full context (no bare numbers)" (docs/METHODOLOGY.md, "Full-context reporting"): every number this skill emits MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare tok/s / TPOT / BW / %SoL / speedup is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, bench backend, serving engine.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named.
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

If this skill produces a measurement (tok/s, latency, %SoL, speedup), follow the rigor discipline: capture L1 zymtrace + L3 DCGM (L4 ncu where feasible) Speed-of-Light and publish --strict. Canonical map: docs/METHODOLOGY.md. Skills that do not produce measurements are exempt (docs/METHODOLOGY.md "Speed-of-light framing").

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per the Grind Mandate (docs/METHODOLOGY.md, "Always be grinding"). A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.