Crispdm phase 4 modeling

CRISP-DM Phase 4 — Modeling (the Analytical Core of a Data Pipeline)

1. Reframing — Phase 4 here is NOT machine learning

Read this once and let it bind every subsequent step.

In the standard CRISP-DM telling (Chapman et al., 2000), Phase 4 sounds like "train an ML model": select modeling technique, generate test design, build model, assess model. That vocabulary maps cleanly onto classification, regression, and clustering — and almost nothing else. Two decades later, the most consequential analytical outputs in AEC, urban, environmental, and public-policy data pipelines are not predictive ML models. They are:

• composite indicators (barrier-severity index, liveability score, vulnerability index),
• scoring maps (per-block, per-parcel, per-patch),
• spatial diagnostics (hotspots, LISA clusters, autocorrelation maps),
• network connectivity products (criticality ranks, percolation curves, restoration priorities),
• rule-based classifications (typology assignments by expert-stated rules),
• simulations (counterfactual / scenario outputs).

This skill treats Phase 4 expansively. Modeling, in the pipeline sense, is the step where prepared data becomes the analytical product the project promised in Phase 1. It is the analytical core — the place where domain logic, normalization choices, weighting decisions, spatial assumptions, and graph constructions are encoded into a reproducible computation. The CRISP-DM tasks (Select Technique → Generate Test Design → Build Model → Assess Model) still structure the work; what changes is the catalogue of valid techniques and the assessment criteria.

Two failures this reframing is built to prevent:

1. Reaching for ML when data, ground truth, sample size, and downstream use do not justify it (Rudin, 2019). Many "ML" problems are composite-indicator or rule-based problems in disguise.
2. Choosing a non-ML technique correctly, then under-investing in methodological rigour — treating normalization, weights, thresholds, and aggregation as cosmetic preprocessing rather than load-bearing analytical decisions (OECD/JRC, 2008).

If you find yourself thinking "I'll just train a random forest", stop. First answer: what question is the pipeline answering? The question selects the family — not the other way around.

---

2. Hard precondition — Phase 3 YAML data contract MUST exist

Phase 4 is downstream of Phase 3 (Data Preparation). Before starting any work in this skill, confirm that a Phase 3 data contract exists for every input the model will consume. The contract is a YAML (or equivalent structured) document recording:

• file path, vintage, row count, primary key, geometry CRS,
• column dtypes, units, missingness pattern,
• known limitations from the Phase 3 datasheet,
• the join keys this dataset uses to bridge to others.

If no Phase 3 data contract exists, halt. Do not improvise it; return the user to Phase 3. Phase 4 built on undocumented inputs becomes irreproducible the moment a column changes. Most "Phase 4 failures" are actually unresolved Phase 3 problems surfacing late.

---

3. Required inputs — the skill refuses to proceed without all five

Block any Phase 4 work until the user can state:

1. The analytical question — exactly one sentence, in canonical form:

   "For each [unit of analysis], the pipeline outputs [a number / a class / a connection / a probability / a forecast] that represents [concept]."

   The unit of analysis must be explicit (block, parcel, patch, person, sensor reading, edge). If the user cannot produce this sentence, refuse to start. There is no "what model should I use" without "what question are you answering" — that question is the entire skill.

2. The decision the output will inform — who acts on this result, and how? If no one, the model has no audience and Phase 4 is premature.

3. The Phase 1 success criterion — what would make this model a successful answer to the question? Stated in non-modeling language.

4. The prepared dataset(s) — path, vintage, row count, key columns, and the Phase 3 data contract (see §2).

5. Confirmation that this is a pipeline project, not an ML training project — this changes the dominant test-design family from train/test splits to sensitivity analysis. If the user is actually doing ML research, route them to a different skill.

If any of (1)–(5) is missing or fuzzy, recover it before proceeding. Most Phase 4 disasters trace back to skipping this gate.

---

Specialist lenses

Codex runs single-agent — there is no sub-agent spawning here. Where the original workflow routed a step to a specialist, adopt that specialist's lens inline: reason through the step as that expert would before writing the artifact. For an independent second opinion (e.g. Phase 5 verification), run the step in a fresh Codex session primed with the specialist role instead.

Curated for Phase 4:

• AI Engineer — ML routes when applicable, model design + deployment integration
• Analytics Reporter — composite indicator design, dashboard logic, KPI translation
• Model QA Specialist — sensitivity analysis, calibration testing, interpretability audit
• Financial Analyst — weighted-index methodology (AHP, budget allocation, weighting schemes)
• Data Engineer — feature pipeline + spatial integration when models depend on it

---

4. Decision tree — route the question to the right technique family

Walk the user through this exact text tree. Terminal nodes route to a build block in §5. Copy-paste this into the user's modeling notes.

What is the pipeline's output for each unit?

|-- A RANKING on a multi-dimensional concept
|   (barrier severity, liveability, vulnerability, opportunity)
|   --> COMPOSITE INDICATOR
|       Route 4A. Canonical: OECD/JRC (2008); Nardo et al. (2005); Booysen (2002).
|
|-- A CLASS LABEL from explicit, expert-stated rules
|   (zoning typology, risk band by code, eligibility flag)
|   --> RULE-BASED CLASSIFICATION
|       Route 4B. Canonical: Russell & Norvig (2021, ch. 9).
|
|-- A CLASS LABEL or VALUE learned from many labelled past cases
|   (prediction with ground truth at sufficient size + representativeness)
|   --> STATISTICAL / ML MODEL
|       Route 4C. Canonical: Hastie, Tibshirani & Friedman (2009); Rudin (2019).
|       **Check the ML gate first (below). Most "ML" routes fail it.**
|
|-- A SPATIAL PATTERN — clusters, hotspots, autocorrelation
|   (where are the unusual concentrations?)
|   --> SPATIAL DIAGNOSTIC (LISA / Gi*)
|       Route 4D. Canonical: Anselin (1995); Getis & Ord (1992).
|
|-- A VALUE at UNOBSERVED LOCATIONS
|   (interpolation surface from point samples)
|   --> GEOSTATISTICAL INTERPOLATION
|       Route 4E. Canonical: Cressie (1993); Hengl et al. (2007).
|
|-- A STRUCTURAL PROPERTY of the system
|   (connectivity, criticality, percolation, betweenness)
|   --> NETWORK / GRAPH ANALYSIS
|       Route 4F. Canonical: Newman (2003); Urban & Keitt (2001); Saura & Pascual-Hortal (2007).
|
`-- A COUNTERFACTUAL — "what would happen if...?"
    (scenario, agent-based dynamics, diffusion)
    --> SIMULATION
        Route 4G. Canonical: Railsback & Grimm (2019).

ML gate. Before routing to 4C, confirm all three:

• A labelled ground truth exists, with enough samples and representativeness for the population of interest.
• The decision is genuinely a prediction (not a ranking dressed as a prediction).
• The downstream user accepts opaque outputs — OR you can use an interpretable model class (Rudin, 2019).

If any of these fail, route back to 4A (composite) or 4B (rule-based). Many problems labelled "ML" are actually composite-index or rule-based problems wearing a costume.

Multiple cores are normal. A single project (the Barcelona mycorrhizal pipeline is a worked example) can legitimately have one composite-indicator core AND one network-analysis core. Treat them as distinct analytical cores, each with its own §5 build, §6 test, and §8 model card. Do NOT fuse them into a mega-score — see anti-patterns §10.

See references/technique-decision-tree.md for the long-form decision tree with worked routing examples.

---

5. Route-specific build steps

Pick the block matching the decision-tree terminal node. Steps 1, 6, 7, 8, 9, 10 (below in §6–§11) are shared across all routes.

Route 4A — Composite indicator (the most common pipeline route)

Anchor: OECD/JRC (2008) Handbook on Constructing Composite Indicators. This is the gold-standard methodology. Do not invent composite-index methodology; follow the handbook and document any deviation.

1. Indicator set. Each indicator gets: definition, source (Phase 3 data contract), vintage, direction (does higher = "more" of the concept?). Reverse-coded indicators stated explicitly.
2. Normalization. Default: min–max to [0, 1] with min/max set at the 5th and 95th percentiles (winsorized) — robust against outliers. State the choice and the alternatives (z-score, ranking, distance-to-reference, categorical) (OECD/JRC, 2008, ch. 4). The choice can flip rankings — it is a substantive analytical decision, not a preprocessing detail.
3. Weighting. Default is NOT equal weighting. Equal weighting is a substantive claim about indicator importance that must be defended (OECD/JRC, 2008, p. 31). Preferred schemes:
   • AHP (Saaty, 1980, 2008) — pairwise expert comparison with consistency ratio reported.
   • Budget Allocation Process — experts distribute a fixed point budget.
   • PCA / factor analysis — only if you want statistical (not substantive) importance, and only as a sensitivity check, not as the headline. If equal weighting is unavoidable, document why AND run AHP or PCA as a sensitivity comparison.
4. Aggregation. Choose:
   • Linear (weighted sum) — fully compensatory; weak dimensions are offset by strong ones.
   • Geometric (weighted product) — partially compensatory; weak dimensions drag the score down (UNDP HDI uses this since 2010).
   • Non-compensatory (MULTIMOORA, PROMETHEE, ELECTRE) — no trade-off; for genuinely incommensurable dimensions. Choice must follow from substantive theory of the concept.
5. Compute. Store per-unit indicator values, normalized values, weights, and final score in a flat, joinable, hash-able table. No hidden steps. No code-embedded magic numbers — all params in a config file.

See references/composite-indicators.md for the OECD/JRC ten-step synthesis (normalization, weighting, aggregation, uncertainty/sensitivity).

Route 4B — Rule-based classification

1. Externalize the rules into a decision-table file (CSV, YAML, or markdown) that a domain expert can read and edit without touching code. Each rule: IF condition THEN class.
2. Coverage. Every unit gets classified by exactly one rule — or by an explicit "uncovered" class. No silent fallbacks.
3. Hold-out expert labelling. Test the ruleset on a held-out batch of expert-labelled units; report disagreements; refine.
4. Version the rule table. Hash inputs, tag outputs.

Route 4C — Statistical / ML model (ONLY after passing the ML gate)

1. Prefer the most interpretable model class that meets the performance bar (Rudin, 2019). Start with logistic regression, GAMs, decision trees, or constrained linear models. Move to ensembles only if necessary, and only if interpretability is recoverable post-hoc.
2. Splits respect data structure. Spatial blocks for spatial data (not random row splits — they leak). Time-respecting splits for time series.
3. Tune on validation only; touch test exactly once.
4. Document features, target, splits, hyperparameters, random seeds. Reproducibility is binary.
5. Bias testing across demographic groups where humans are involved. Fairness metrics reported alongside performance.

Route 4D — Spatial diagnostic (LISA / Gi*)

1. Spatial weights matrix. Choose (rook, queen, k-nearest, distance band) and justify. State this explicitly; "Moran's I with no weights statement" is an unacceptable artefact (Anselin, 1995).
2. Compute Local Moran's I (Anselin, 1995) or Getis-Ord Gi* (Getis & Ord, 1992).
3. Multiple-testing correction. Apply false discovery rate (Caldas de Castro & Singer, 2006). Without correction, hotspot maps over-detect.
4. MAUP (Openshaw, 1984). Rerun at a second spatial unit and compare. If results are unstable, that is the finding — report it.
5. Edge effects. State how units near the study boundary were treated (buffered area, edge weighting, omission).

See references/spatial-techniques.md for the deep dive (zonal stats, LISA, Gi*, MAUP, edge effects, spatial autocorrelation as diagnostic).

Route 4E — Geostatistical interpolation

1. Inspect sample design. Clustered samples bias the prediction surface — flag this before fitting anything.
2. Variogram. Fit; choose kriging variant (ordinary, universal, regression-kriging with covariates per Hengl et al., 2007).
3. Leave-one-out cross-validation. Report RMSE in source units, not just a number.
4. Produce BOTH the prediction surface AND the uncertainty surface. Never one without the other — an interpolated map without uncertainty is a lie.

Route 4F — Network / graph analysis

1. Graph construction decisions are the model. Write one paragraph each for: what is a node? what is an edge? what is the edge weight? what is the existence threshold? (Urban & Keitt, 2001.)
2. Metrics. Component analysis; Probability of Connectivity (Saura & Pascual-Hortal, 2007); centrality (degree, betweenness, current-flow betweenness per Newman, 2005); percolation curve (Stauffer & Aharony, 1994).
3. Threshold sensitivity. Edge thresholds and distance-decay parameters MUST be swept — at least three values each. Report the curve, not the point. Identify which nodes/edges are robustly critical across the grid.
4. Construct validity. Compare critical-node ranking to independently identified critical patches (expert panel, prior literature).

See references/network-techniques.md for the deep dive (graph construction as modeling choice, connectivity metrics, threshold sensitivity).

Route 4G — Simulation

1. Behavioural primitives. Specify what each agent / state does at each timestep, and which of those primitives you can validate against data.
2. Ensemble. At least 100 stochastic replicates. Report distributions, not single trajectories.
3. Honesty. A simulation is an argument, not a prediction, unless calibrated against ground truth (Railsback & Grimm, 2019). State this in the model card under "out-of-scope".

---

6. Pre-registered test design (write BEFORE the final build)

Open test-design.md and pre-register the tests before finalizing the build. Post-hoc selection of favourable tests is cherry-picking — the cardinal sin of Phase 4.

For every route, the test design covers at minimum:

• Sensitivity grid (4A, 4D, 4F primarily): which choices vary, over what values, how rank/score variation will be summarised. Variance-based methods (Saltelli et al., 2008, 2010) where feasible; otherwise full-factorial scenario runs.
• Alternative-specification comparison (every route): a defensibly different choice at the most consequential methodological fork — run it and report.
• Expert / stakeholder face validation (every route): which experts, what they see, what counts as agreement. Structured protocol > ad-hoc.
• Ground-truth comparison where possible: hold-out samples for composite indices; independently flagged critical patches for network models; in-situ measurements for interpolation surfaces.
• Internal consistency checks: Cronbach's alpha within sub-dimensions for composite indices (Nardo et al., 2005, ch. 5); centrality rank stability under edge resampling for networks.
• Cross-data validation: rerun on a parallel dataset (different year, different city, different sensor) where available; compare structural results.

CRISP-ML(Q) (Studer et al., 2021) extends CRISP-DM with explicit quality-assurance tasks at every phase. The discipline applies to non-ML cores too.

See references/sensitivity-analysis.md for Saltelli methods (Sobol' indices, Morris screening, minimum viable sensitivity protocols).

After the build, execute every pre-registered test. Append results to test-design.md. Never silently drop tests that produced unfriendly results.

---

7. Model assessment criteria for non-ML analytical cores

For pipeline cores, "accuracy" is often the wrong frame. There is no held-out truth; there is only defensibility to a critical reader. Assess on:

• Robustness. Do substantive conclusions survive reasonable variation in normalization, weighting, aggregation, spatial weights, edge thresholds? Quantify via the sensitivity grid (Saltelli et al., 2008).
• Stability. Do conclusions survive small input perturbations (noise injection, jackknife on observations, alternative imputation)?
• Interpretability. Can a domain expert reconstruct, in plain words, why a particular unit got its score? If not, prefer a simpler model (Rudin, 2019).
• Internal consistency. Within-dimension indicators correlate as theory predicts; sub-indices correlate with the overall index appropriately.
• Construct validity. Does the model measure the concept it claims to? Expert review, comparison to established proxies, convergent/discriminant validity (Nardo et al., 2005, ch. 5).
• Communicability. A unit whose rank flips across the sensitivity grid should be communicated as flipping, not as belonging to one tier. Honesty about uncertainty is part of the output.
• Defensibility. Could you, in front of a critical Phase 5 audience, defend every choice? If not, document the gap.

Write the assessment into model-card.md. State explicitly which conclusions are robust and which are fragile.

---

8. Model card template — Mitchell et al. (2019) adapted

Every analytical core gets a model card modelled on Mitchell et al. (2019), adapted for non-ML analytical artefacts. Minimum sections:

1. Purpose and intended use
2. Out-of-scope uses (what this model must NOT be used for)
3. Input data summary (with pointer to Phase 3 datasheet)
4. Modeling technique — family, specific method, references
5. Parameter and choice log — every normalization / weight / aggregation / threshold, one paragraph each
6. Test design and assessment results
7. Robustness statement — which conclusions are robust, which are fragile
8. Interpretability statement — how a non-author can reconstruct any unit's score
9. Known limitations and ethical considerations — MAUP, edge effects, equity implications, missing populations
10. Versioning — input hashes, code git SHA, parameter config hash
11. Authors and reviewers

See references/model-card-template.md for the fillable template.

Companion artefacts:

• Decision log — prose record of the order in which methodological choices were made and the alternatives considered.
• Weight justification table (composite-indicator routes only) — every weight mapped to its source (literature / expert panel / statistical procedure) with a one-line rationale.

Datasheets for Datasets (Gebru et al., 2021) cover the upstream data and should already exist from Phase 3.

---

9. Anti-patterns (fail the work if you see these)

See references/anti-patterns.md for the full catalogue and remediations. Highlights:

1. Black-box composite. A single number with no breakdown. (Booysen, 2002.)
2. Equal weighting by default with no justification. Equal weighting is a substantive claim, not the absence of one. State it AND defend it. (OECD/JRC, 2008, p. 31.)
3. Hidden normalization. Switching from min–max to z-score silently flipped a ranking. Both must be documented and reported.
4. No sensitivity analysis. The single most common Phase 4 failure. Without it, the composite or network is a position statement, not a result. (Saltelli et al., 2008.)
5. Mistaking precision for accuracy. Four-decimal scores from a fragile composite. Report only as many significant digits as sensitivity supports.
6. Single edge threshold in a network. Report the curve. (Urban & Keitt, 2001.)
7. ML where a rule would do. Random forest for a four-feature decision a domain expert solves in their head. (Rudin, 2019.)
8. Spatial weights unspecified. Moran's I or Gi* with no statement of weight construction. (Anselin, 1995.)
9. Cherry-picked specification. Many runs, one reported. Fix: pre-registered test design.
10. No model card. No surviving reproducible artefact = no pipeline.
11. Composite + network fused into a mega-score. They answer different questions. Keep them separate.
12. "Validated by face validity" with no protocol. Document which experts, what they saw, what counted as agreement.

---

10. Exit criteria — 8 artefacts before handoff to Phase 5

Phase 4 is NOT done until ALL eight exist. If any is missing, do not move to Phase 5.

1. Model card(s) — one per analytical core, per §8.
2. Sensitivity analysis log — every {normalization × weighting × aggregation × threshold × imputation} run, with rank/score outputs and a summary robust-tier classification.
3. Decision log — prose record of the order and rationale of methodological choices, with alternatives considered.
4. Weight justification table (composite routes only) — every weight mapped to source with rationale.
5. Construct-validity evidence — expert panel notes, independent dataset comparisons, convergent/discriminant checks.
6. Reproducible artefact — versioned code (git SHA), hashed inputs, parameters in a config file (not hard-coded), and a single-command script that regenerates the headline output AND the sensitivity outputs.
7. Out-of-scope statement — explicit list of uses the model is NOT fit for.
8. Open questions and known limitations — MAUP, edge effects, missing populations, contested weights, tests that could not be performed.

Quality gate — refuse to declare Phase 4 complete unless every checkbox is yes:

• One-sentence analytical question written and matches the actual output?
• Technique family justified against rejected alternatives?
• All methodological choices logged with paragraph-length rationale?
• Test design pre-registered before the final build?
• Sensitivity analysis run across at least three forks (normalization, weighting, aggregation — or their network-analysis equivalents)?
• Robust-tier / robust-rank classification produced — NOT a single point estimate?
• At least one external/expert validity check?
• Reproducible: one command regenerates everything?
• Model card, sensitivity log, weight-justification table written?
• Out-of-scope uses stated?
• Equity considerations (whose data is missing, who bears the cost of error) considered?

If any answer is no, Phase 4 is not done.

---

11. Handoff to Phase 5

Phase 5 (Evaluation) tests the model not just technically but against the Phase 1 mission objectives (Chapman et al., 2000, ch. 5). For the review to be honest rather than ceremonial, the eight artefacts above must cross from Phase 4 to Phase 5 before the review meeting — not during it.

Phase 5 reviewers will check:

• Does the model card's "intended use" match the Phase 1 question?
• Does the sensitivity log support the headline robustness claim?
• Are the out-of-scope statements consistent with how the output is being communicated?
• Have the construct-validity checks been performed by people independent of the modellers?

If artefacts are missing, Phase 5 degenerates into either uncritical acceptance or undeserved dismissal — both are pipeline-design failures.

---

12. References

Full bibliography in references/citations.md. Core anchors:

• Anselin, L. (1995). Local indicators of spatial association — LISA. Geographical Analysis, 27(2), 93–115.
• Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59(2), 115–151.
• Chapman, P. et al. (2000). CRISP-DM 1.0: Step-by-Step Data Mining Guide. SPSS / The CRISP-DM consortium.
• Gebru, T. et al. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
• Getis, A., & Ord, J. K. (1992). The analysis of spatial association by use of distance statistics. Geographical Analysis, 24(3), 189–206.
• Mitchell, M. et al. (2019). Model cards for model reporting. FAT '19*, 220–229.
• Nardo, M. et al. (2005). Tools for Composite Indicators Building. JRC, EUR 21682 EN.
• Newman, M. E. J. (2003). The structure and function of complex networks. SIAM Review, 45(2), 167–256.
• OECD & JRC (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. OECD Publishing. (Gold standard for composite indicators.)
• Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
• Saaty, T. L. (2008). Decision making with the analytic hierarchy process. International Journal of Services Sciences, 1(1), 83–98.
• Saltelli, A. et al. (2008). Global Sensitivity Analysis: The Primer. Wiley.
• Studer, S. et al. (2021). Towards CRISP-ML(Q). Machine Learning and Knowledge Extraction, 3(2), 392–413.
• Urban, D., & Keitt, T. (2001). Landscape connectivity: A graph-theoretic perspective. Ecology, 82(5), 1205–1218.