Mlops observability

Skill MLOps-Courses/mlops-coding-skills/mlops-observability

Agent skills based on the MLOps Coding Course

Install

npx -y skills add MLOps-Courses/mlops-coding-skills --skill mlops-observability

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

19 stars19 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Guide to implement full stack observability including reproducibility, lineage, monitoring, alerting, and explainability.

SKILL.md

2.8 KB, as published. Nobody here has run it

MLOps Observability

Goal

To implement a "Glass Box" system where every result is Reproducible, every asset has Lineage, and system health is Monitored, Alerted on, and Explained.

Prerequisites

Language: Python
Context: Production monitoring and debugging.
Platform Suggestion: MLflow, SHAP, Evidently, ...

Instructions

1. Guarantee Reproducibility

Consistency is key. For instance:

Randomness: Set seeds for random, numpy, torch, tensorflow.
Environment: Use docker and locked dependencies (uv.lock).
Builds: Use mise run build (wrapping uv build --build-constraint) for deterministic wheels.
Code: Track git commit hash for every run.

2. Track Data Lineage

Know the origin of your data. For instance:

Datasets: Create MLflow Datasets with mlflow.data.from_pandas.
Logging: Log inputs to MLflow context with mlflow.log_input.
Versioning: Version data files (e.g., data/v1.csv) or use DVC.
Transformations: Log preprocessing parameters mapping data versions to model versions.

3. Monitoring & Drift Detection

Watch for silent failures. For instance:

Validation: Gate models against quality thresholds with mlflow.validate_evaluation_results (MLflow 3).
Drift: Use evidently to compare reference (training) vs current (production) data.
- Detect Data Drift (input distribution changes) and Concept Drift (relationship changes).
System: Enable MLflow System Metrics (log_system_metrics=True) for CPU/GPU.

4. Alerting

Don't stare at dashboards. For instance:

Local: Use plyer for desktop notifications during long training runs.
Production: Use PagerDuty (critical) or Slack (warnings).
Thresholds: Use Static (fixed value) or Dynamic (anomaly detection) rules.
Action: Alerts must link to a dashboard or playbook.

5. Explainability (XAI)

Trust but verify. For instance:

Global: Use Feature Importance (e.g., Random Forest) to understand overall logic.
Local: Use SHAP values to explain individual predictions.
Artifacts: Save explanations (plots/tables) as MLflow artifacts.

6. Infrastructure & Costs

Optimize resources. For instance:

Tags: Tag runs with project, env, user.
Costs: Log run_time and instance type to estimate ROI.

Self-Correction Checklist

Seeds: Are random seeds fixed?
Inputs: Are input datasets logged to MLflow?
System Metrics: Is log_system_metrics enabled?
Explanations: Are SHAP values generated?
Alerts: Are thresholds defined for failures?