Trackio

Skill boeschj/ml-intern-plugin/skills/trackio

Configure trackio experiment monitoring for training runs, add alert callbacks, read alerts back between runs, and derive the next run's config from prior alerts. Use for any training job monitoring or when iterating on hyperparameters.From its SKILL.md

Install

npx -y skills add boeschj/ml-intern-plugin --skill trackio

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.7 KB, 642 tokens by cl100k_base, as published. Nobody here has run it

Trackio is natively integrated with Transformers Trainer and all TRL trainers; the built-in TrackioCallback handles init, log, and finish. In TrainingArguments, SFTConfig, DPOConfig, or GRPOConfig set:

report_to="trackio"
run_name="sft_qwen3-4b_lr2e-5_bs128"
project="<descriptive-project-name>"
trackio_space_id="<namespace>/ml-intern-a1b2c3d4"

Replace the bracketed values with real ones before the script leaves your hands: the namespace comes from hf-whoami --brief, the project groups related runs for comparison, and the space id suffix is any 8 characters. TRACKIO_PROJECT and TRACKIO_SPACE_ID env vars work too. Always give the user the dashboard URL: https://huggingface.co/spaces/<trackio_space_id>.

Alerts drive iteration

Use trackio.alert(title, text, level) at every decision point in training. Levels:

ERROR: stop and change approach (divergence, NaN, OOM)
WARN: tweak hyperparameters (overfitting, early stopping, KL spike, reward collapse, slow convergence)
INFO: milestones (training complete, target reached, checkpoint saved)

Always include numeric values and an actionable suggestion in the text, for example "loss=12.4 at step 200, lr likely too high, try x0.1". A future run must be able to parse it and act.

Wire alerts through a custom TrainerCallback passed via callbacks=[...]: on_log for training metrics (loss, reward, kl), on_evaluate for eval metrics (only available there). Keep each condition simple: one metric, one threshold.

Reading alerts back

Read alerts between runs instead of parsing thousands of metric points. CLI, always with --json:

trackio get alerts --project <p> --run <r> --json
trackio get alerts --project <p> --since <iso8601> --json
trackio get run    --project <p> --run <r> --json
trackio get metric --project <p> --run <r> --metric <m> --json
trackio list runs  --project <p> --json

Python: api = trackio.Api(); api.alerts(p, run=r, since=ts); api.runs(p) where each run carries .name, .config, .alerts().

The plugin's job monitor watches job states and failure logs for you; read alerts with the CLI between runs and react to ERROR alerts immediately instead of waiting for the job to finish.

Next config from prior alerts

diverged: lr x 0.1
overfitting: weight_decay x 10 or reduce capacity
early stopping: lr x 0.5 or adjust the schedule
high accuracy: refine around the current config

Read the prior config via api.runs(...).config and only mutate keys the alerts justify changing.

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.