Explainability

Explainability Skill

Purpose

A saliency / Grad-CAM heat-map is the most over-interpreted artifact in medical-imaging AI: a colourful map over the lesion is routinely presented as proof the model "looks at the right thing." Adebayo et al. (NeurIPS 2018) showed many saliency methods produce visually convincing maps that are independent of the model's learned weights and of the labels — so they explain nothing. This skill produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).

It sits alongside evaluation in the lane: /architecture-zoo → /preprocess-imaging → /model-scaffold → /model-validation → /model-evaluation + explainability → /write-paper + /check-reporting. It integrates captum / pytorch-grad-cam (referenced in the plan); it does not reimplement them and never runs a model on real patient data.

When to use

• You produced (or are about to produce) saliency / Grad-CAM / attention maps and want them reported to the standard a reviewer expects.
• You want to audit an explainability analysis for the four failure modes below.

When NOT to use

• Discrimination / calibration metrics → /model-evaluation then /analyze-stats.
• Split or preprocessing leakage → /model-validation / /preprocess-imaging.
• LLM/MLLM faithfulness & hallucination → /mllm-eval.
• Reimplementing captum / pytorch-grad-cam → out of scope (this skill wires and audits them).

The four failure modes (what the gate enforces)

1. Saliency as validation. A map is attribution, not proof the model is correct or that the relationship is causal. Frame it as "where signal is attributed", never as "the model is right".
2. No sanity check. Run the Adebayo model-randomisation and data-randomisation tests. A map that survives neither is uninterpretable; both axes are the minimum bar.
3. No quantitative localisation. If you claim the map localises the finding, measure it — IoU / pointing game / Dice against ground-truth masks — do not eyeball a few examples.
4. Cherry-picked examples. Report a cohort-level result, not a handful of hand-picked cases.

Workflow

Phase 1 — Produce the maps (integrate, don't reimplement)

Choose the method for the architecture (references/explainability_guide.md): Grad-CAM / Grad-CAM++ for CNNs, attention-rollout for ViTs, integrated-gradients / SHAP for attribution. Wire captum or pytorch-grad-cam; do not write a new CAM implementation.

Phase 2 — Sanity-check and quantify

• Run the model-parameter randomisation and data (label) randomisation tests (Adebayo 2018); a faithful map degrades when the model/labels are randomised.
• Compute a quantitative localisation metric against ground-truth masks (IoU / pointing game / Dice) over the cohort — not a visual impression.

Phase 3 — Emit the explainability-report manifest

{
  "method": "grad-cam++",
  "n_examples": 200,
  "cohort_level": true,
  "localization_metric": "iou",
  "localization_value": 0.63,
  "sanity_checks": ["model_randomization", "data_randomization"],
  "interpretation": "localization"
}

interpretation: attribution / localization / faithfulness (descriptive) — never validation / causal (overclaim).

Phase 4 — Gate the report (deterministic)

python3 scripts/check_explainability_report.py --manifest explainability_report.json --strict

Verdicts: SALIENCY_AS_VALIDATION, NO_SANITY_CHECK, NO_LOCALIZATION_METRIC (Major); INSUFFICIENT_SANITY, CHERRY_PICKED_EXAMPLES, MISSING_METHOD (Minor). The verdict is reproduced by rule on the manifest, never asserted from prose.

Integration

• /model-evaluation — explainability accompanies the held-out metrics as a secondary analysis.
• /self-review ai_overclaiming / image_synthesis probes audit saliency overclaiming in a finished manuscript; this skill produces the rigorous analysis they look for.
• /check-reporting — the manifest documents the CLAIM 2024 / TRIPOD+AI interpretability items.

Anti-Hallucination

• Never fabricate saliency maps, localisation metrics, or sanity-check results. Every value in the manifest comes from the researcher's executed XAI code — never invented. This skill designs and audits the analysis; it does not run a model on real patient data.
• Never present a saliency map as proof of model correctness or causation. A map is attribution; claiming it validates the model is the overclaim this skill exists to prevent (SALIENCY_AS_VALIDATION).
• Never report an explainability-audit "pass" without running check_explainability_report.py. The rigor verdict is reproduced deterministically, never asserted from prose.
• Integrate, don't reimplement. Reference captum / pytorch-grad-cam; do not write a new CAM / attribution implementation or claim results for one.

Reproducible challenge

scripts/check_explainability_report_challenge/ ships a synthetic weak/strong report pair with a network-free verify.sh wired into the skill's validation commands.