agentsclimarketplace

Drift monitoring

Skill Amey-Thakur/AI-SKILLS/skills/machine-learning/drift-monitoring

Plug-and-play skills and prompts for every AI coding agent

Install
npx -y skills add Amey-Thakur/AI-SKILLS --skill drift-monitoring

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
  • 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Detect input, prediction, and performance drift with reference windows and act through retrain or rollback triggers. Use when operating models in production or diagnosing gradual quality decay.

SKILL.md

3.4 KB, as published. Nobody here has run it

Drift monitoring

Models decay silently: the world changes, inputs shift, and accuracy erodes long before anyone complains. Monitoring layers three signals by label latency, because ground truth is usually late or missing.

Method

  1. Monitor performance directly where labels arrive. Rolling metric on recent labeled outcomes (the same metric and slices as the offline eval; see model-evaluation), against the shipped baseline. This is the truth; the other layers exist because truth is often days-to-never late (label latency mapped in ml-problem-framing).
  2. Watch prediction drift as the early warning. Score/prediction distribution vs a reference window (launch period or trailing stable month): population stability index, mean score, positive rate. A fraud model whose flag rate doubles overnight is telling you something broke upstream or the world moved; it fires days before labeled metrics can.
  3. Watch input drift per feature, ranked by importance. Null rates, out-of-range values, category share shifts, distribution distances (PSI/KS) on the top features (see feature-engineering). Sudden input drift is usually a pipeline bug (schema change, broken join, unit change: see schema-evolution) wearing a statistics costume; check data quality before blaming the world (see data-quality-checks).
  4. Choose references and thresholds deliberately. Fixed reference (training distribution) detects total drift; trailing reference detects sudden change while tolerating slow drift: run both. Calibrate alert thresholds on historical variance (seasonality is not drift; weekends are not incidents), start warn-only for two weeks, and slice drift by segment: aggregate stability can hide one region on fire (see data-quality-checks alert discipline).
  5. Bind alerts to actions in advance. Input anomaly: page the data owner, check pipelines. Prediction drift past X: investigate, consider threshold re-tuning (base-rate moves; see imbalanced-data). Performance below the floor: trigger the retrain runway or roll back to a previous model (see model-deployment). A drift dashboard without an action table is weather reporting.
  6. Close the loop with scheduled evaluation. Even absent alerts, re-evaluate on fresh labeled data monthly/quarterly and retrain on a cadence justified by measured decay speed, not by habit; each retrain travels the full gated deployment path, and its win is verified against the incumbent (see ml-baselines, ab-test-design).

Boundaries

  • Drift detection flags change, not cause; concept drift (the input-output relationship moved) needs retraining, while covariate shift sometimes only needs threshold recalibration: diagnose via ml-error-analysis before spending the retrain.
  • Feedback loops (the model's own actions shape future data: lending, ranking) bias every monitored signal; where stakes justify it, hold out a small randomized slice as an unbiased measurement channel.
  • Retraining on drifted-and-unlabeled data via pseudo-labels compounds errors; do not automate retrain-on-drift without a labeled gate.

Gives 0 of the 12 instructions most monitoring observability skills give

Counted across 481 of the 483 authors here whose files we hold, read 2026-08-06

  • link every alert to a runbookin 43 of 481, across 35 files
  • use structured json loggingin 36 of 481, across 31 files
  • alert on user-facing symptomsin 20 of 481, across 15 files
  • emit structured JSON logs with stable event namesin 18 of 481, across 13 files
  • propagate trace context across boundariesin 16 of 481
  • use histograms for latency trackingin 14 of 481, across 9 files
  • use OpenTelemetry for distributed tracingin 13 of 481, across 8 files
  • include a correlation ID on every log linein 13 of 481, across 8 files
  • Define service level objectivesin 10 of 481, across 7 files
  • Call useAzureMonitor before importing other modulesin 9 of 481, across 2 files
  • stop and ask for clarification if inputs are missingin 9 of 481, across 2 files
  • define on-call questions before adding telemetryin 9 of 481, across 4 files

Said here and by no other author read

  • monitor performance against shipped baseline where labels exist
  • watch prediction distribution versus a reference window
  • rank input drift by feature importance
  • run both fixed and trailing reference windows
  • calibrate alert thresholds on historical variance
  • slice drift by segment

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.