Image algorithm validator

Image Algorithm Validator Skill

Purpose

The Image Algorithm Validator Skill supports validation of medical image processing algorithms, including segmentation, detection, and analysis algorithms, ensuring performance meets clinical requirements.

Capabilities

• Ground truth dataset curation guidance
• Performance metric calculation (Dice, IoU, sensitivity, specificity)
• Inter-observer variability analysis
• Statistical comparison methods
• Validation dataset stratification
• Multi-reader multi-case study design
• FDA AI/ML guidance alignment
• Failure case analysis
• Edge case identification
• Performance boundary testing
• Cross-validation methodology

Usage Guidelines

When to Use

• Validating image analysis algorithms
• Curating validation datasets
• Designing reader studies
• Preparing regulatory submissions

Prerequisites

• Algorithm development complete
• Ground truth established
• Validation dataset available
• Performance criteria defined

Best Practices

• Use representative, diverse datasets
• Establish robust ground truth methodology
• Assess performance across subgroups
• Document failure modes

Process Integration

This skill integrates with the following processes:

• Medical Image Processing Algorithm Development
• AI/ML Medical Device Development
• Clinical Evaluation Report Development
• Software Verification and Validation

Dependencies

• SimpleITK library
• scikit-image
• MONAI framework
• Evaluation frameworks
• Statistical analysis tools

Configuration

image-algorithm-validator:
  algorithm-types:
    - segmentation
    - detection
    - classification
    - registration
    - quantification
  metrics:
    - Dice
    - IoU
    - sensitivity
    - specificity
    - AUC
    - Hausdorff-distance
  validation-methods:
    - holdout
    - cross-validation
    - external-validation

Output Artifacts

• Dataset curation protocols
• Ground truth documentation
• Performance reports
• Statistical analyses
• Reader study results
• Failure mode catalogs
• Regulatory submission sections
• Validation summaries

Quality Criteria

• Ground truth methodology validated
• Metrics appropriate for algorithm type
• Dataset representative of intended use
• Statistical analysis rigorous
• Subgroup performance assessed
• Documentation supports regulatory review