Sota ml engineering

SOTA ML Engineering / MLOps (2026)

Expert rules for building and auditing production machine-learning systems — the lifecycle that turns a model into a reliable, monitored, governed service. This is classical/predictive ML (tabular, ranking, vision, forecasting, recommendation): training pipelines, feature stores, model registries, serving, and drift monitoring. It is not LLM-application engineering — prompts, RAG, agents, and LLM evals live in sota-llm-engineering; data pipelines/warehouses live in sota-data-engineering. Grounded in Google's Rules of ML, the ML Test Score rubric, and Hidden Technical Debt in ML Systems. Every rule states the why; every rules file ends with an audit checklist.

Purpose

Two consumers, one source of truth:

• BUILD mode — building ML systems: follow the rules as defaults. The model is a small part; the system around it (data, features, serving, monitoring, governance) is where production ML succeeds or rots.
• AUDIT mode — reviewing an ML system: hunt violations with the audit checklists, classify by severity, report in the finding format below. Train/serve skew, data leakage, and an unmonitored model in production are presumed-serious until disproven.

BUILD mode

1. Before building, read the rules files relevant to the task (see index). A new model service needs 01, 02, 04, 05, 06.
2. Apply the top-10 non-negotiables (below) unconditionally.
3. Start simple (Rules of ML #1: don't be afraid to launch a product without ML; then a simple model with a solid pipeline beats a fancy model on a broken one). Build the pipeline, metrics, and monitoring first; the model is iterated inside that frame.
4. Make everything reproducible and versioned — data, features, code, config, model, environment — so any model in production can be rebuilt and explained.
5. Guarantee training/serving consistency: the same feature transformations at train and inference time (a feature store or shared transform code), or you will ship train/serve skew (Rules of ML #29, #31, #32).
6. When you take a shortcut (manual step, un-versioned data, no slice metrics), leave a # NOTE(sota): and a tracking item — ML technical debt compounds silently.

AUDIT mode

Work each relevant rules file's audit checklist against the system: the training pipeline, the feature/serving path, the registry, and the monitoring. The ML Test Score (data / model / infra / monitoring tests) is the backbone rubric — score each category. Confirm claims against the code and pipeline config, not the diagram.

Severity conventions

Severity	Meaning	Examples
CRITICAL	Silently wrong predictions in production, or exploitable	Data leakage inflating offline metrics, train/serve skew on the prediction path, label leakage, deserializing an untrusted pickle/model, no rollback for a bad model
HIGH	Likely incident or unsafe deployment	No drift/performance monitoring in prod, no validation gate before deploy, non-reproducible model (can't rebuild), unversioned data/features, no slice metrics on a high-stakes model, PII in features without basis
MEDIUM	Correctness/maintainability hazard / debt	Single aggregate metric only, no baseline, manual deploy steps, feature computed two ways, no experiment tracking, glue-code/pipeline-jungle, undeclared consumers of a model output
LOW	Debt that will bite later	Unused features kept in infra, no model card, notebook-only training, weak naming/versioning hygiene
INFO	Style/doc/hygiene	Missing docstrings, dashboard polish, minor config sprawl

Finding format

[SEVERITY] path:LINE (or pipeline stage) — short title
  Rule: rules/NN-name.md § section
  Evidence: code/config/metric, verbatim
  Impact: one sentence — what predicts wrong / fails / leaks, under what condition
  Fix: concrete change or control
  Effort: trivial | small | medium | large

Group by severity, CRITICAL first. End with: counts per severity, an ML Test Score-style readiness summary (data/model/infra/monitoring), and the three highest-leverage fixes.

Rules index

File	Read this when...
rules/01-ml-systems-architecture.md	Designing/reviewing an ML system: the model-is-small-part principle, training vs serving paths, feature store, model registry, reproducibility, the Hidden-Technical-Debt anti-patterns (entanglement/CACE, glue code, pipeline jungles, undeclared consumers, feedback loops)
rules/02-data-and-features.md	Anything touching training data or features: data leakage and label leakage, train/serve skew, feature/data versioning, splits (temporal/group), feature engineering discipline, dropping unused features, PII minimization
rules/03-training-experimentation.md	Training and iterating: experiment tracking & reproducibility (seeds, env, data hash), hyperparameter search, distributed training/checkpointing, config management, reproducible runs, starting simple
rules/04-evaluation-validation.md	Deciding if a model is good enough: offline metrics vs the business objective, baselines, sliced evaluation and fairness, the ML Test Score tests, validation gates and regression thresholds before promotion
rules/05-deployment-serving.md	Shipping a model: packaging (containers/ONNX), batch vs online vs streaming serving, model registry promotion, canary/shadow/A-B rollout, rollback, latency/throughput, reproducible inference environment
rules/06-monitoring-drift.md	Operating a model: data drift (PSI/KS) vs concept drift vs performance decay, label lag, prediction & feature monitoring, alerting, retraining triggers and cadence, ML-specific observability (cross-ref sota-observability)
rules/07-security-governance.md	ML security & compliance: training-data poisoning, model extraction/inversion/membership inference, adversarial inputs, supply chain (untrusted pickle/model artifacts, dataset provenance), MITRE ATLAS, NIST AI RMF, model cards, EU AI Act obligations

Top-10 non-negotiables

1. No data leakage. No information from the target or the future or the test set enters training features (no fitting scalers/encoders on the full dataset before split, no post-outcome features). Leakage inflates offline metrics and is CRITICAL — it makes a broken model look great. (rules/02, rules/04)
2. No train/serve skew. The exact feature transformations used in training are used at inference — shared code or a feature store, not reimplemented twice. Verify with skew checks. (rules/01, rules/02)
3. Everything is reproducible and versioned — data, features, code, config, environment, and the model artifact — so any production model can be rebuilt and explained. A model you can't reproduce is HIGH. (rules/01, rules/03)
4. Pipeline and monitoring before model sophistication. A simple model on a solid, monitored pipeline beats a fancy model on a fragile one (Rules of ML). (rules/01)
5. Evaluate on slices and against a baseline, not one aggregate number. Report per-segment metrics and fairness-relevant slices; a model that wins on average can fail badly on a subgroup. (rules/04)
6. A validation gate guards promotion. Automated checks (metric thresholds, no regression vs current prod, slice floors, data/schema validation) must pass before a model is promoted; deploys are reversible with fast rollback. (rules/04, rules/05)
7. Production models are monitored for drift and decay. Data drift (PSI/KS), prediction distribution, and — as labels arrive — live performance, with alerts and a retraining trigger. An unmonitored model silently rots. (rules/06)
8. Never deserialize an untrusted model/pickle. pickle/joblib/torch.load on an untrusted artifact is arbitrary code execution; verify provenance and integrity (hashes/signing), prefer safe formats (safetensors, ONNX). (rules/07)
9. Govern data and the model. Minimize PII in features and document a lawful basis; produce a model card; map risks with MITRE ATLAS / NIST AI RMF; check EU AI Act obligations for high-risk use. (rules/07)
10. Kill ML debt deliberately. Drop unused features, delete dead pipelines, untangle glue code, declare consumers of model outputs, and break feedback loops — the Hidden-Technical-Debt anti-patterns. (rules/01)