Ml model validation regulatory

Skill rbr7/MedClawMini/skills/ml-model-validation-regulatory

A focused, production-minded library of 197 clinical-AI and healthcare data-science skills for the OpenClaw agent platform featuring data quality, clinical NLP, big-data ML, explainable AI, drug safety, and regulatory.

Install

npx -y skills add rbr7/MedClawMini --skill ml-model-validation-regulatory

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Validate, document, and monitor clinical machine-learning models for governance and regulatory readiness (Software as a Medical Device / SaMD). Covers validation protocols, model cards and intended-use statements, Good Machine Learning Practice (GMLP), IEC 62304 software lifecycle and ISO 14971 risk management for AI, performance and subgroup acceptance criteria, and post-deployment drift/performance monitoring. Use to take a trained model from notebook to a documented, auditable, monitored production asset.

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

4.9 KB, as published. Nobody here has run it

ML Model Validation & Regulatory Readiness

Overview

A model that influences care is a regulated, risk-bearing asset. This skill wraps the engineering of healthcare-predictive-modeling and explainable-ml-healthcare in the validation, documentation, and monitoring that governance and regulators expect turning a trained model into an auditable, maintainable production system aligned with GMLP, IEC 62304, and ISO 14971.

When to Use This Skill

Promoting a clinical/operational model from notebook to documented production.
Preparing model documentation for internal governance, audit, or regulatory submission.
Defining acceptance criteria and a validation protocol before deployment.
Standing up post-deployment monitoring for drift and performance decay.

Components

Intended use & risk intended-use statement, target population, role in the clinical workflow; ISO 14971 risk analysis of failure modes and mitigations.
Validation protocol pre-specified datasets (internal + external/temporal), acceptance thresholds (overall and per subgroup), and the analyses to run written before seeing test results.
Performance validation discrimination, calibration, and decision-curve/net- benefit on held-out and external data; subgroup fairness against thresholds.
Robustness sensitivity to missingness, input shifts, and edge cases; stress tests.
Documentation a complete model card + datasheet: data lineage, training, metrics, limitations, intended use, and explainability summary.
Lifecycle (IEC 62304) versioning of data/code/model, change control, and a defined update/retraining procedure (and where applicable a Predetermined Change Control Plan).
Monitoring production dashboards for input drift (PSI/KS), prediction drift, and live performance vs. acceptance thresholds, with alerts and a rollback plan.

Example

# Pre-specified validation gate (pass/fail against thresholds)
import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.calibration import calibration_curve

def validate(y, p, groups, min_auroc=0.75, max_calib_err=0.10, min_subgroup_auroc=0.70):
    report = {"auroc": roc_auc_score(y, p)}
    frac_pos, mean_pred = calibration_curve(y, p, n_bins=10)
    report["calibration_error"] = float(np.mean(np.abs(frac_pos - mean_pred)))
    report["subgroups"] = {g: roc_auc_score(y[idx], p[idx]) for g, idx in groups.items()}
    report["PASS"] = (report["auroc"] >= min_auroc
                      and report["calibration_error"] <= max_calib_err
                      and all(v >= min_subgroup_auroc for v in report["subgroups"].values()))
    return report

# Production drift monitor (Population Stability Index per feature)
def psi(expected, actual, bins=10):
    q = np.quantile(expected, np.linspace(0,1,bins+1))
    e = np.histogram(expected, q)[0]/len(expected) + 1e-6
    a = np.histogram(actual,   q)[0]/len(actual)  + 1e-6
    return float(np.sum((a-e)*np.log(a/e)))     # alert if > 0.25

Outputs

validation_protocol.md (pre-registered) and validation_report.md (results vs. gate).
model_card.md + datasheet.md governance-ready documentation.
risk_analysis.md ISO 14971-style hazard/mitigation table.
monitoring_plan.md + a drift/performance dashboard spec with alert thresholds.

Healthcare Context

Bridges data science and the medical-device quality world a natural complement to the repo's iso-13485-certification and hipaa-compliance skills. Encodes the non-negotiables for clinical AI: external/temporal validation, calibration, subgroup fairness, and live monitoring. This is documentation and process tooling, not legal advice; involve your regulatory/quality function for actual submissions.

References

FDA/Health Canada/MHRA Good Machine Learning Practice (GMLP); FDA SaMD & AI/ML guidance.
IEC 62304 (medical device software lifecycle); ISO 14971 (risk management).
Sendak et al. (2020), Presenting Machine Learning Model Information to Clinical End Users (model facts labels).