Hf jobs

Discipline for HF cloud compute. The hf-jobs executable is the only submission path; a PreToolUse guard prices every run and a background monitor reports state changes and failures back to you, so submitted jobs never need manual polling loops.

Authenticated namespace: !"${CLAUDE_PLUGIN_ROOT}/bin/hf-whoami" --brief

Current hardware catalog (USD per hour):

"${CLAUDE_PLUGIN_ROOT}/bin/hf-jobs" hardware 2>/dev/null || echo '{"ok": false, "error": "catalog unavailable, run hf-jobs hardware manually"}'

Submission contract

hf-jobs run refuses to submit unless all of these hold, so do them in order:

1. --timeout is explicit. Set it from model size, minimum 2h for any training. The 30 minute default has killed real training runs; that is why the flag is mandatory.
2. --confirm-cost matches or exceeds the estimate for billable hardware. Get the number first: hf-jobs estimate --flavor a10g-large --timeout 4h.
3. The script contains no placeholder sentinels (<username>, <model-name>, and friends). Resolve the real namespace before writing the script.

Never pass a local machine path as the job script reference. Jobs run in a fresh cloud environment where local files do not exist. Submit inline source (--inline reads stdin), a --script whose content is uploaded, or a public --url.

For a job that needs a specific container rather than a UV script, swap the source for --image <docker-image> --command "<cmd>"; the same --timeout and --confirm-cost rules still apply, and it works for immediate and scheduled runs alike.

Exact-source workflow

For any non-trivial job script:

1. Write the script locally with Write.
2. Validate syntax and imports.
3. Smoke test the exact script: one training step plus one eval step when evaluation is part of the workflow. GPU code paths (CUDA, bf16/fp16, quantization, flash attention kernels, torch.compile, model loading) need a GPU smoke test: hf-sandbox create --hardware t4-small minimum, Ampere or newer (never T4) when a flash-attention kernel is involved. GPU sandboxes are billable, so hf-sandbox create refuses billable hardware without --confirm-cost <USD> at or above the estimate, exactly like hf-jobs run; free cpu-basic needs no confirmation. CPU sandboxes cannot test GPU paths.
4. Submit the exact tested source. Do not reconstruct a similar script from memory.

If you skip the GPU smoke test, state why before running the job.

Every training script fails fast before expensive work:

• print package versions for torch, transformers, trl, accelerate, datasets, trackio, and kernels when used
• assert required dataset columns exist
• assert push_to_hub=True and hub_model_id is set with no placeholder
• include every imported third-party package in --dep, plus the core stack: torch, transformers, trl, accelerate, datasets, trackio, and kernels~=0.12.0 when using Hub kernels

Preflight, stated before every run

• Reference implementation: [which example this is based on]
• Dataset format verified: [columns confirmed via hf-dataset-inspect]
• GPU sandbox smoke test: [hardware and result, or why not applicable]
• push_to_hub and hub_model_id: [set, no placeholders]
• timeout: [value] based on [model size] on [hardware]
• estimated cost: [from hf-jobs estimate]
• trackio: [space id, public dashboard]

Cannot fill an item: stop and complete it first.

Hardware sizing

• 1-3B params: a10g-largex2
• 7-13B params: a100-large
• 30B+ params: l40sx4 or a100x4
• 70B+ params: a100x8

a10g-small and a10g-large have the same 24GB GPU memory; the difference is CPU and RAM only.

Batches, scheduled jobs, failures

Ablations and sweeps: submit ONE job, watch the monitor confirm it reaches training, then submit the rest. Prefer one sweep script over ten manual submissions.

Scheduled jobs always require human approval; their recurring cost is unbounded. Auto permission modes refuse them by design; ask the user to approve in default mode.

On failure the monitor prints the log tail. Diagnose from hf-jobs logs <id>, fix the actual cause, and follow the OOM ladder from /ml-intern rather than shrinking the task. Cancelling: hf-jobs cancel <id>; the ledger reconciles actual cost when the terminal state lands.