Ai ml engineering

Overview

ML engineering failures are silent and delayed. A model that scores well on the benchmark can fail badly in production. This skill enforces the practices that catch these failures before they reach users: proper evaluation harnesses, data leakage detection, distribution shift monitoring, and safety checks.

When to Use

• Before training or fine-tuning a model
• Before deploying a model to production
• When integrating a third-party LLM API
• When evaluating model quality
• When debugging unexpected model behavior

Process

Step 1: Define the task and success metric precisely

Before any code: what is the exact prediction task? What metric proves the model is good enough? What metric proves it is safe enough? Document these as your evaluation contract.

Step 2: Establish the baseline

Compute a simple baseline (majority class, rule-based system, GPT-4 zero-shot). Your model must beat this baseline by a meaningful margin to justify the complexity.

Step 3: Audit the training data

• Check for data leakage (test set information in training set)
• Check for label quality (sample 100 examples manually)
• Check for demographic skew (does the dataset represent production distribution?)
• Check for PII that should not be in training data
• Document the data provenance and version

Step 4: Implement a reproducible training pipeline

• Pin all dependency versions
• Set all random seeds
• Version the dataset (not just the model)
• Store training hyperparameters with model artifacts
• Confirm: can you reproduce this exact model from scratch?

Step 5: Build the evaluation harness before training

Write your evaluation pipeline before training. Evaluations should be:

• Automatic (run in CI)
• Deterministic (same inputs → same scores)
• Multi-dimensional (accuracy, latency, cost, safety, fairness)
• Comprehensive (held-out test set + edge case suite)

Step 6: Train with monitoring

Track: training loss, validation loss, gradient norms. Flag: loss spikes, NaN gradients, overfitting (train loss << val loss), underfitting.

Step 7: Run the full evaluation suite

Compare against: baseline, previous model version, human performance (if applicable). Document every dimension. Declare the threshold required for deployment.

Step 8: Safety evaluation

For LLM applications:

• Test for prompt injection
• Test for jailbreak attempts
• Test for harmful output generation
• Test for PII leakage in outputs
• Test for hallucination in factual claims

Step 9: Production readiness

• Latency profiled at p50, p95, p99
• Cost per inference calculated
• Graceful degradation defined (what happens when the model is unavailable?)
• Output validation implemented (reject malformed outputs)
• Monitoring in place for distribution shift
• Feedback loop defined for collecting production labels

Step 10: Staged rollout

Deploy to 1% of traffic. Monitor key metrics for 24 hours. Roll out to 10%, then 100%. Have a rollback procedure.

Anti-Rationalizations

"The eval numbers look good" Eval numbers on a curated test set are necessary but not sufficient. Production distribution ≠ test distribution.

"We'll add safety checks after launch" Safety issues discovered after launch are incidents. Safety checks added before launch are requirements.

"The model improved so we should ship it" Improved on which metric? Under which conditions? Improvements in accuracy can come with regressions in latency, fairness, or safety.

Red Flags

• No baseline comparison
• Evaluation dataset overlaps with training data
• Evaluation written after training (to justify the result)
• No monitoring for distribution shift
• Model deployed without latency profiling
• No rollback procedure

Verification Requirements

• Task and success metric defined before training
• Baseline computed and documented
• Training data audited for leakage and PII
• Training is reproducible from version-pinned code + versioned dataset
• Evaluation harness built before training
• Safety evaluation completed
• Latency profiled at p95
• Staged rollout plan documented
• Distribution shift monitoring in place