Explainability by default

Explainability by Default

Every AI system you build must be able to explain its decisions. This is not an add-on — it is a design constraint. Build explanation capability first, then the model.

The Explainability Design Question

Before choosing a model architecture, answer this:

Who needs to understand this decision, and what do they need to know?

Audience	What They Need	Approach
Affected citizen	"Why was I declined/flagged/scored?"	Plain-English local explanation
Frontline staff	"What drove this score for this person?"	Feature importance for this case
Auditor / regulator	"How does this system work systematically?"	Global model behaviour + documentation
Developer / data scientist	"Is the model behaving as expected?"	SHAP values, attention maps, partial dependence

Design for all relevant audiences before writing model code.

---

Step 1: Choose an Interpretable-First Architecture

Prefer interpretable models unless you have a clear, documented reason not to:

Decision tree → Logistic regression → Linear SVM
     ↓ (only if accuracy genuinely requires it)
Random Forest + SHAP → Gradient Boosting + SHAP
     ↓ (only if task genuinely requires it)
Deep neural network + explanation layer
     ↓ (last resort — requires extra governance)
Black-box model (requires independent audit)

Rule: Use the simplest model that meets your accuracy requirements. Document why you didn't use a simpler one.

---

Step 2: Add Explanation Capability at Build Time

For Scikit-learn Models

import shap

# Train model
model.fit(X_train, y_train)

# Build explainer at training time — not as an afterthought
explainer = shap.TreeExplainer(model)  # for tree-based models
# or
explainer = shap.LinearExplainer(model, X_train)  # for linear models
# or
explainer = shap.KernelExplainer(model.predict, shap.sample(X_train, 100))  # for any model

# Save explainer alongside model
import joblib
joblib.dump({'model': model, 'explainer': explainer}, 'model_artifacts/model_with_explainer.pkl')

For Neural Networks

import shap
import torch

# Use DeepExplainer or GradientExplainer
explainer = shap.DeepExplainer(model, background_data)

# Or use integrated gradients via Captum (PyTorch)
from captum.attr import IntegratedGradients
ig = IntegratedGradients(model)

For LLMs / Generative Models

# Implement structured chain-of-thought that surfaces reasoning
SYSTEM_PROMPT = """
When making any classification or recommendation, you MUST:
1. State your conclusion
2. List the top 3 factors that led to it, in order of importance
3. State what would have changed your conclusion
4. Assign a confidence level: High / Medium / Low

Format:
CONCLUSION: <your decision>
FACTORS: 1. <most important> 2. <second> 3. <third>
COUNTERFACTUAL: <what would change the outcome>
CONFIDENCE: <High/Medium/Low> — <reason>
"""

---

Step 3: Implement the Explanation API

Every model must expose a standard explanation interface:

class ExplainableModel:
    """Wrapper that enforces explainability as a first-class concern."""
    
    def __init__(self, model, explainer, feature_names: list[str]):
        self.model = model
        self.explainer = explainer
        self.feature_names = feature_names
    
    def predict(self, X):
        """Make prediction."""
        return self.model.predict(X)
    
    def explain(self, X_single, top_n: int = 5) -> dict:
        """
        Explain a single prediction.
        
        Returns:
            {
                'prediction': <value>,
                'confidence': <float>,
                'top_factors': [
                    {'feature': str, 'value': any, 'impact': float, 'direction': str},
                    ...
                ],
                'plain_english': str,
                'counterfactual': str,
            }
        """
        prediction = self.model.predict(X_single)[0]
        shap_values = self.explainer.shap_values(X_single)
        
        # Get top N features by absolute SHAP value
        importances = list(zip(
            self.feature_names,
            X_single[0],
            shap_values[0] if isinstance(shap_values, list) else shap_values[0]
        ))
        importances.sort(key=lambda x: abs(x[2]), reverse=True)
        
        top_factors = [
            {
                'feature': feat,
                'value': val,
                'impact': float(shap),
                'direction': 'increases' if shap > 0 else 'decreases',
            }
            for feat, val, shap in importances[:top_n]
        ]
        
        plain_english = self._generate_plain_english(prediction, top_factors)
        
        return {
            'prediction': prediction,
            'confidence': float(abs(shap_values).max()),
            'top_factors': top_factors,
            'plain_english': plain_english,
        }
    
    def _generate_plain_english(self, prediction, top_factors) -> str:
        """Generate a plain-English explanation a non-technical person can understand."""
        factor_text = ', '.join(
            f"{f['feature']} ({f['direction']} the score)"
            for f in top_factors[:3]
        )
        return (
            f"The system predicted '{prediction}'. "
            f"The main factors were: {factor_text}."
        )
    
    def global_explanation(self) -> dict:
        """Return global feature importance for the entire model."""
        raise NotImplementedError("Implement global explanation for audit purposes.")

---

Step 4: Test Your Explanations

Explanations can be wrong. Test them:

def test_explanation_is_consistent(model, test_df):
    """Explanation must be consistent with the prediction."""
    for _, row in test_df.sample(20).iterrows():
        result = model.explain(row.to_frame().T)
        # Top positive factor should increase prediction
        top_factor = result['top_factors'][0]
        assert result['prediction'] is not None
        assert len(result['top_factors']) > 0
        assert result['plain_english'] != ""

def test_explanation_identifies_known_driver(model, test_data_with_known_driver):
    """When a feature is the dominant driver, explanation must surface it."""
    # Create a test case where one feature clearly dominates
    result = model.explain(test_data_with_known_driver)
    assert result['top_factors'][0]['feature'] == 'known_dominant_feature', (
        f"Expected known driver to be top factor, got: {result['top_factors'][0]['feature']}"
    )

---

Step 5: Document for Stakeholders

For each deployed system, produce:

1. Global explanation summary — which features matter most overall, and why
2. Example explanations — 5 real examples covering different prediction outcomes
3. Plain-English system description — what the system does, in language suitable for a citizen
4. Limitations statement — when the explanation may be unreliable or incomplete

Add these to docs/explainability-<system-name>.md.

---

Completion Checklist

• Explanation approach chosen and documented before model training
• Explainer built and saved alongside model
• explain() method implemented and tested
• Plain-English explanation generated and reviewed by a non-technical person
• Global explanation documented
• Explanation limitations documented
• Tests written to verify explanation consistency

Proceed to governance-documentation before deployment.