Responsible data handling

Responsible Data Handling

Data is the foundation of AI. How you handle it determines whether the system is lawful, fair, and trustworthy. Before writing a single line of data pipeline code, complete this skill.

Step 1: Classify Your Data

Classify every data source before touching it:

Class	Description	Examples	Requirements
Public	No personal data, publicly available	ONS statistics, OpenStreetMap	Standard care
Internal	Organisation data, no personal info	Product logs (anonymised), config	Access controls
Personal	Identifies or can identify individuals	Names, emails, IP addresses	UK GDPR applies
Sensitive Personal	Special category under UK GDPR	Health, ethnicity, religion, biometrics	Explicit legal basis required
High Risk	Personal + significant impact decisions	Benefits data, criminal records	DPA 2018 Schedule 1 + DPIA

Action: Label every dataset in your code:

# DATA CLASSIFICATION: Sensitive Personal (health records)
# LEGAL BASIS: Article 9(2)(h) — medical treatment purposes
# RETENTION: 8 years per NHS Records Management Code
# ACCESS: Restricted to authorised clinical staff only
df_patients = load_patient_records(...)

---

Step 2: Apply Data Minimisation

Load only what you need. Every unnecessary field is a liability:

# BAD: Loading everything and filtering later
df = pd.read_csv('patients.csv')
df = df[['age', 'diagnosis']]  # the name and postcode were loaded into memory

# GOOD: Select columns at load time
df = pd.read_csv('patients.csv', usecols=['age', 'diagnosis'])

# GOOD: SQL — select only what's needed
query = "SELECT age, diagnosis FROM patients WHERE cohort = 'study_group'"
df = pd.read_sql(query, conn)

Rule: If a column is not in your model's feature set or your analysis plan, do not load it.

---

Step 3: Handle Personal Identifiers

Anonymisation vs Pseudonymisation

Technique	What It Does	Still Personal Data?
Anonymisation	Irreversibly removes all identifiers	No
Pseudonymisation	Replaces identifiers with tokens	Yes — UK GDPR applies
Aggregation	Reports only group statistics	No (if k≥5)

import hashlib
import pandas as pd

def pseudonymise(df: pd.DataFrame, id_cols: list[str], salt: str) -> pd.DataFrame:
    """
    Replace identifying columns with consistent pseudonymous tokens.
    The salt must be stored securely and separately from the data.
    """
    df = df.copy()
    for col in id_cols:
        df[col] = df[col].apply(
            lambda x: hashlib.sha256(f"{salt}{x}".encode()).hexdigest()[:16]
            if pd.notna(x) else None
        )
    return df

# Load salt from environment — NEVER hardcode
import os
SALT = os.environ['DATA_PSEUDONYMISATION_SALT']
df = pseudonymise(df, id_cols=['patient_id', 'nhs_number'], salt=SALT)

Suppression — Small Number Suppression

Never report statistics on groups smaller than 5:

def suppress_small_groups(
    df: pd.DataFrame,
    group_col: str,
    value_col: str,
    min_count: int = 5
) -> pd.DataFrame:
    """Suppress statistics where group count < min_count."""
    counts = df.groupby(group_col)[value_col].count()
    suppressed_groups = counts[counts < min_count].index
    df = df.copy()
    df.loc[df[group_col].isin(suppressed_groups), value_col] = None
    return df

---

Step 4: Secure Data Storage and Access

# NEVER store credentials in code
# BAD:
conn = connect(host='db.internal', password='mypassword123')

# GOOD: Environment variables or secrets manager
import os
conn = connect(
    host=os.environ['DB_HOST'],
    password=os.environ['DB_PASSWORD']
)

# NEVER log personal data
import logging
logger = logging.getLogger(__name__)

# BAD:
logger.info(f"Processing patient {patient_name}, DOB {date_of_birth}")

# GOOD:
logger.info(f"Processing patient record {record_id}")

File permissions — never leave data world-readable

# Set appropriate permissions on data files
chmod 640 data/sensitive_dataset.csv  # owner + group read, no world access
chmod 700 data/raw/                    # only owner can access raw directory

---

Step 5: Audit Logging

Every access to personal or sensitive data must be logged:

import logging
import json
from datetime import datetime, timezone

audit_logger = logging.getLogger('audit')

def log_data_access(
    user: str,
    dataset: str,
    purpose: str,
    records_accessed: int,
    fields_accessed: list[str]
) -> None:
    """Write an immutable audit log entry for data access."""
    entry = {
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'user': user,
        'dataset': dataset,
        'purpose': purpose,
        'records_accessed': records_accessed,
        'fields_accessed': fields_accessed,
    }
    audit_logger.info(json.dumps(entry))

# Use it everywhere you access personal data
log_data_access(
    user=current_user(),
    dataset='patient_records',
    purpose='training_diagnostic_classifier',
    records_accessed=len(df),
    fields_accessed=list(df.columns)
)

---

Step 6: Data Retention and Deletion

Every dataset must have a documented retention schedule:

# Add to every data loading function
DATA_RETENTION = {
    'patient_records': '8 years (NHS Records Management Code)',
    'benefits_applications': '6 years (Limitation Act 1980)',
    'model_training_data': '3 years (internal policy)',
    'audit_logs': '7 years (HMRC / public sector standard)',
}

def check_retention(dataset_name: str, data_date: str) -> None:
    """Raise if dataset is past its retention date."""
    from dateutil.parser import parse
    from dateutil.relativedelta import relativedelta
    ...

---

Completion Checklist

• Every data source classified (Public / Internal / Personal / Sensitive / High Risk)
• Legal basis documented in code comments for any personal data
• Data minimisation applied — only necessary columns loaded
• Personal identifiers pseudonymised or anonymised as appropriate
• Small number suppression applied to any published statistics
• No credentials or personal data in logs or code
• Audit logging implemented for all personal data access
• Retention schedule documented

Proceed with your data pipeline. Run bias-assessment before training any model on this data.