Crispdm phase 5 evaluation

CRISP-DM Phase 5 — Evaluation

The orienting sentence. Phase 4 asks: did the model work? Phase 5 asks: did the pipeline answer the question the decision-maker actually has, in a form they can act on, with caveats they can defend? Phase 5 starts only after the Phase-1 business success criteria are on the table, and ends only after a named decision-maker has said, on the record, whether they would use the output on Monday.

This skill is the final reality check before a CRISP-DM pipeline is allowed to leave the workshop. It is brutally skeptical by design. Its default verdict is NEEDS WORK, and it requires overwhelming, named, dated, file-backed evidence to flip to SHIP.

---

1. Phase 4 vs Phase 5 — the distinction this skill exists to enforce

The single most common error in CRISP-DM use is collapsing Phase 4 ("Modeling") and Phase 5 ("Evaluation") into one undifferentiated "did the model work?" stage. The CRISP-DM 1.0 reference guide is unusually explicit that these are different activities, with different audiences, different success criteria, and different artifacts.

Phase 4 — Assess Model (Chapman et al., 2000, p. 28):

"the data mining engineer interprets the models according to his domain knowledge, the data mining success criteria and the desired test design… he judges the success of the application of modeling and discovery techniques more technically."

Audience: the engineer themselves. Success criteria: the data-mining success criteria from Phase 1 (AUC ≥ 0.8, RMSE ≤ X, runtime < N minutes). Output: a model assessment and revised parameter settings.

Phase 5 — Evaluate Results (Chapman et al., 2000, p. 30):

"Previous evaluation steps dealt with factors such as the accuracy and generality of the model. This step assesses the degree to which the model meets the business objectives and seeks to determine if there is some business reason why this model is deficient."

Audience: the business — the urban planner, the scientist, the policymaker, the operations team that will (or will not) act on the output. Success criteria: the business success criteria from Phase 1. Outputs: an "Assessment of data mining results with respect to business success criteria", a list of "Approved models", a Review of Process, a List of Possible Actions, and a Decision (Chapman et al., 2000, pp. 30–31).

In pipeline projects — composite rasters, ranked priority maps, alert systems, dashboards — there is often no model in the supervised-learning sense at all. A Barcelona urban-ecology pipeline that composites NDVI, soil maps, OSM street-tree records, and DEM-derived connectivity into a "barrier-reduction priority map" passes through no train/test split. There is nothing to "assess" in Phase-4 terms. But it absolutely requires Phase 5: does this map answer the question an urban planner actually asked, at the resolution they can act on, with caveats they can defend?

If you cannot articulate this distinction in one sentence before starting, you are not yet doing Phase 5.

See references/phase-4-vs-5-distinction.md for verbatim Chapman 2000 quotes and the pipeline-relevant elaboration.

---

2. The anti-pattern this skill prevents

The model scored well, therefore we ship.

Or, in pipeline form:

The map looks plausible, the script runs end-to-end, therefore we ship.

Both fail because nobody asked whether the output answers the Phase-1 business question in a form the named decision-maker would actually use. Technical success is necessary but not sufficient. The empirical literature on data-science failure (Sambasivan et al., 2021; Sculley et al., 2015; Heger et al., 2022) is consistent: pipelines that pass Phase-4 metrics routinely fail in production because Phase-5 evaluation was skipped, performed by the team in isolation, or replaced with vibes.

Every step below exists to block that anti-pattern. See references/anti-patterns.md for the full catalogue.

---

3. Hard precondition — refuse to start without these

This skill refuses to begin if either of the following is missing. Stop, name what's missing, and send the user back to Phase 1.

Required before Step 0:

1. Phase-1 business success criteria, verbatim. Look for, in order:

   • phase-1-business-understanding/success-criteria.md
   • phase-1/business-criteria.md
   • business-understanding/criteria.md
   • A criteria block at the top of README.md
   • User-pasted text labelled "Phase-1 business success criteria"

   If none are present, respond verbatim:

   Phase 5 evaluates against the Phase-1 business success criteria. Without those criteria written down, this is theatre. Go produce them — return when you can paste them in. (Chapman et al., 2000, p. 30.)

2. A cancellation criterion. A written statement of the condition(s) under which the project would be killed. Without this, the Go/Iterate/Kill decision degenerates into "ship at any cost" — sunk-cost defence dressed as evaluation (Kahneman, 2011). Look for:

   • phase-1-business-understanding/cancellation-criterion.md
   • A "we would kill this project if…" section in the success-criteria file
   • User-pasted text labelled "cancellation criterion"

   If absent, respond:

   Phase 5 cannot produce a defensible Go / Iterate / Kill decision without a pre-written cancellation criterion. Write one (one sentence: "we would stop this project if X"), then return.

Do not negotiate these preconditions. They are the load-bearing artifacts the rest of the skill assumes.

---

4. Required inputs

Once the preconditions are met, gather:

1. The Phase-1 business question — one sentence, the decision the pipeline is meant to inform.
2. The named decision-maker — institution and role (e.g., "Ajuntament de Barcelona, Urban Ecology Department, Director"). "TBD" is not acceptable.
3. A pinned, runnable version of the pipeline — commit hash, environment file (lockfile), raw-input manifest with file hashes.
4. A list of the pipeline's outputs — raster, table, ranked list, alert, dashboard URL — and where each lives on disk.
5. Access to at least one named representative of the decision-maker for the stakeholder walkthrough (live, recorded interview, or a summary that the representative has reviewed and signed off on).
6. (Optional) Phase-4 model-assessment outputs, prior data documentation (datasheets), sensitivity-analysis outputs, independent reproduction logs.

Missing items 1–5: refuse to proceed. Missing item 6: note as a Phase-5 finding (gap in evidence).

---

Specialist lenses

Codex runs single-agent — there is no sub-agent spawning here. Where the original workflow routed a step to a specialist, adopt that specialist's lens inline: reason through the step as that expert would before writing the artifact. For an independent second opinion (e.g. Phase 5 verification), run the step in a fresh Codex session primed with the specialist role instead.

Curated for Phase 5:

• Reality Checker — evidence-based certification, refuses fantasy approvals
• Evidence Collector — screenshot-obsessed, requires visual proof
• Code Reviewer — independent second opinion on pipeline outputs
• UX Researcher — stakeholder walkthrough design + decision-cycle mapping
• Model QA Specialist — independent audit of Phase 4 artifacts (different lens from Phase 4 assessment)

---

5. The eleven sequential steps

Do not skip. Do not reorder. Each step produces an artifact the next step depends on.

Step 0 — Gate check

Re-confirm Section 3 preconditions and Section 4 inputs are all present. List anything missing. Stop if any required item is absent.

Step 1 — Name the Phase 4 / Phase 5 distinction in writing

Open evaluation-report.md with the orienting sentence (Section 1 above) verbatim. This is the line that prevents the most common Phase-5 failure (the assessment/evaluation conflation). It must appear at the top of the report. Cite Chapman et al. (2000, pp. 28, 30).

Step 2 — Close the loop on Phase-1 criteria

For each Phase-1 business success criterion, produce a row in the Evaluation Report:

| Criterion (verbatim) | Status (met / partial / unmet / un-evaluable) | Evidence (file path, line number, screenshot, log) | Residual risk |

un-evaluable is a valid Phase-5 finding, not an escape hatch — it means the criterion was not operationalised well enough to be checked, which is itself an outcome.

No criterion may be marked met without file-backed evidence. Verbal assurance is not evidence. See references/evaluation-report-template.md.

Step 3 — Fitness-for-purpose audit

Run the six-item checklist from Hamilton et al. (2022) and produce yes / no / partial / NA for each, with a one-line consequence per non-yes:

• Does the output answer the original Phase-1 question, or a question the pipeline made convenient?
• Is the spatial / temporal / categorical resolution matched to the decision unit?
• Is the stated confidence honest (real uncertainty propagation, not goodness-of-fit only)?
• Is timeliness adequate for the decision?
• Are data licences, ethical clearances, and re-use rights compatible with intended use?
• Is the pipeline reproducible end-to-end by an independent operator on a clean machine?

Full checklist with elaboration: references/fitness-for-purpose-checklist.md.

Step 4 — Stakeholder walkthrough

Run the script in references/stakeholder-walkthrough-script.md with the named decision-maker in the room (or on a call). Capture verbatim: hesitations, missing-context requests, invented caveats, and the answer to the closing Monday test:

"Would you act on this output on Monday morning, in front of your colleagues, without further work?"

Record the answer in walkthrough-notes.md exactly as spoken. Do not paraphrase. Do not summarise. The verbatim answer is the load-bearing data point of the entire phase.

If no decision-maker is available, stop. Phase 5 without a decision-maker is not Phase 5 (Madaio et al., 2020).

Step 5 — Ethical and bias review (pipeline level)

Open references/ethics-review.md and answer each prompt in plain language:

• Who benefits? Name specific people / institutions.
• Who is missed? Whose phenomena, geographies, or populations are systematically under-represented in the inputs?
• What is the failure-cost asymmetry? Cost of a false positive vs a false negative. Who bears each cost? Are the costs symmetric or skewed?
• What does the pipeline do at the edges of its coverage? How is that signalled to the user?
• Is there a route by which an affected party can contest or correct an output?

For environmental pipelines, the dominant bias is usually sampling bias (dense observations near universities, sparse in peripheral neighbourhoods), not demographic bias in the classical fairness-ML sense. Write the mechanism, not the label.

Step 6 — Process review + pre-mortem

Run a structured pass through every prior phase and produce a four-column register in process-review.md:

• Shortcuts taken (filters without sensitivity analysis, defaults left untouched, single CRS assumed, missing values dropped, default resampling).
• Assumptions not validated.
• Inputs with thin provenance (datasets whose licence / vintage / accuracy could not be confirmed at primary source).
• Steps that should be re-run before deployment.

Then run a pre-mortem (Kahneman, 2011):

"Imagine this pipeline has been live for six months and a journalist or auditor has just shown the decision-maker a serious failure. Write the post-mortem now."

Add every failure mode surfaced in the pre-mortem to the register. Pre-mortem outputs are evidence, not speculation — confirmation bias is the default, and the pre-mortem is the antidote.

Step 7 — Limitations Register

Convert findings from Steps 2–6 into a versioned, line-item file limitations-register.md using the template:

| ID | Limitation | Source phase | Severity (L/M/H) | Trigger condition | Mitigation | Owner |

This file lives in the repository, not in slides. Every limitation must have an Owner (named) and a Trigger condition (the situation under which it becomes a real failure). Limitations without owners are wishes. See references/limitations-register-template.md.

Step 8 — Intended-Use Statement

Following Mitchell et al. (2019), write intended-use-statement.md:

• Pipeline name + version (commit hash).
• Primary intended uses.
• Primary intended users (named institutions / roles).
• Out-of-scope uses (named).
• Prohibited uses (named). For pipelines that could be misused for enforcement, punitive, or appeals-process decisions, this list is mandatory and must be specific.
• Performance characteristics across coverage tiers (where reliability degrades).
• Date of next required re-evaluation.

Step 9 — Conditions for Deployment / Conditions for Non-Use

Two short lists in conditions-deployment.md and conditions-non-use.md:

• Deployment — preconditions that must hold for release (e.g., "second-team reproduction succeeds and hash-matches"; "decision-maker has signed off in writing on the bias caveat").
• Non-Use — contexts where the pipeline must not be used (e.g., "must not be used as sole evidence in planning appeals"; "must not be used outside the Barcelona municipal boundary"; "must not be used after [date] without re-running the land-cover layer").

Both feed directly into Phase 6 — Deployment.

Step 10 — Go / Iterate / Kill decision

Produce go-no-go-memo.md using the template in references/go-no-go-memo-template.md. One page maximum:

• Recommendation: ship / iterate / kill.
• Recommender (named).
• Dissenting opinions (named — or "no dissenting opinions" stated explicitly).
• Rationale, tied to specific Phase-1 criteria and the verbatim Monday-test answer.
• If iterate: the written hypothesis for what will change and which phase to loop back to. Iteration without a hypothesis is sunk-cost defence.
• If kill: explicit acknowledgement that sunk costs do not justify continuing, and a written statement of what future related projects must read this memo for.

Default recommendation is NEEDS WORK (= iterate). SHIP requires all of:

• Every Phase-1 criterion met or explicitly partial-with-written-acceptance by the decision-maker.
• Monday-test answer is "yes" or "yes with caveats" (caveats then enumerated in Conditions for Non-Use).
• Independent reproduction by a second operator on a clean machine has succeeded with hash-matching outputs (or differences explained in writing).
• Limitations Register, Intended-Use Statement, Conditions for Deployment, Conditions for Non-Use are all signed.
• No pre-mortem failure mode is marked Severity = H without a written mitigation.

Anything less → NEEDS WORK. Do not soften.

Step 11 — Phase 6 handoff bundle

Confirm the existence of every artifact required for handoff. Refuse to declare Phase 5 complete if any is missing. See references/evaluation-report-template.md § Handoff for the full bundle list.

---

6. Exit criteria (the 10-item checklist)

All ten must be true before Phase 5 is complete. The Monday test is the ultimate gate.

• 1. The Phase 4 / Phase 5 distinction is stated verbatim at the top of the Evaluation Report, with Chapman 2000 citation.
• 2. Every Phase-1 business success criterion appears verbatim and has a status + file-backed evidence.
• 3. A real decision-maker (not the pipeline team) participated in the walkthrough; their name and role are recorded.
• 4. The Monday-test answer is recorded verbatim, not paraphrased.
• 5. The Limitations Register contains at least one line per Phase-1 criterion marked partial / unmet, each with a named Owner.
• 6. The Intended-Use Statement names prohibited uses, not only intended uses.
• 7. The Go/Iterate/Kill memo names dissenting opinions (or states "no dissenting opinions" explicitly).
• 8. If recommendation is iterate, a written hypothesis for what will change exists.
• 9. If recommendation is ship, an independent operator has re-run the pipeline end-to-end on a clean machine and the result is logged.
• 10. The Monday test gate. No Phase-4 technical metric (AUC, RMSE, accuracy) appears alone as evidence for a Phase-1 criterion without an explicit translation step to the business question — AND the Monday-test answer is on the record and matches the recommendation.

If any of these is false: NEEDS WORK. Do not ship.

---

7. Anti-patterns this skill auto-detects and refuses

In order of seriousness. The first is the founding sin of CRISP-DM practice.

1. Assessment / evaluation conflation. The user offers AUC / RMSE / accuracy / F1 as "the evaluation". Stop. Respond: "That is Phase-4 Assess Model. Phase 5 evaluates against the Phase-1 business success criteria. Show me those." Cite Chapman et al. (2000, pp. 28, 30).
2. No decision-maker in the room. The team is evaluating against criteria the team invented and presenting it to itself. Stop. Require a named representative of the decision-maker for the walkthrough (Madaio et al., 2020).
3. Confirmation bias. The team built the pipeline; they will find it acceptable by default. Require the pre-mortem (Step 6) and treat its outputs as evidence (Kahneman, 2011).
4. Evaluating in isolation from intended use. Evaluation at the team's desk only. Require the decision-rehearsal session in the intended-use context (Voinov et al., 2018).
5. Vanishing limitations. Limitations surfaced verbally and not committed to limitations-register.md. Treat the evaluation as incomplete.
6. Skipping Review Process. Treating Evaluate Results as the whole of Phase 5 and omitting the meta-review. CRISP-DM 1.0 marks these as distinct tasks (Chapman et al., 2000, pp. 30–31).
7. Treating "no" as failure. A clean no-go with a defensible evaluation is a successful Phase 5. Do not push the user toward "ship" to avoid the appearance of failure.
8. Vague Phase-1 criteria. "Provide useful insight to planners" is not falsifiable. Force operationalisation before evaluating against it.

Full catalogue with worked examples: references/anti-patterns.md.

---

8. Handoff to Phase 6 — Deployment

The Go memo is the handoff token. Phase 6 refuses to begin without it. Do not pass anything to Phase 6 unless the recommendation is ship and the full bundle exists:

• evaluation-report.md
• limitations-register.md
• intended-use-statement.md
• conditions-deployment.md
• conditions-non-use.md
• walkthrough-notes.md (with verbatim Monday-test answer)
• process-review.md (with pre-mortem)
• go-no-go-memo.md (signed, named recommender, named dissenters)
• A reproducibility bundle: pinned environment (lockfile), raw-input manifest with file hashes, single run script, expected-output hashes (Pineau et al., 2021).
• A named owner for monitoring and a re-evaluation cadence (calendar date).

If recommendation is iterate, hand back to the named phase with the written hypothesis. Do not loop back without one.

If recommendation is kill, archive the bundle alongside the kill rationale. A clean kill is documentation; do not delete it.

---

9. Output artifacts (write to disk)

All Phase-5 outputs live in evaluation/<pipeline-name>/ at the project root. The skill writes (or updates) these eight files:

1. evaluation-report.md — structured by Phase-1 criteria; orienting Phase-4/5 distinction at the top.
2. walkthrough-notes.md — verbatim notes from the stakeholder walkthrough, including the Monday-test answer.
3. process-review.md — pre-mortem + register of shortcuts and unchecked assumptions.
4. limitations-register.md — versioned, line-item, every limitation has an Owner.
5. intended-use-statement.md — model-card-style; prohibited uses named.
6. conditions-deployment.md — bullet list of preconditions for release.
7. conditions-non-use.md — bullet list of contexts where the pipeline must not be used.
8. go-no-go-memo.md — one page, signed (named).

Plus the Phase-6 handoff bundle (reproducibility manifest + owner + cadence) when recommendation = ship.

---

10. References

Chapman, P., Clinton, J., Kerber, R., Khabaza, T., Reinartz, T., Shearer, C., & Wirth, R. (2000). CRISP-DM 1.0: Step-by-step data mining guide (esp. pp. 28–31). SPSS / The CRISP-DM Consortium.

Hamilton, S. H., et al. (2022). A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental Modelling & Software, 148, 105278.

Heger, A., Marquis, L. B., Vorvoreanu, M., Wallach, H. M., & Vaughan, J. W. (2022). Understanding machine learning practitioners' data documentation perceptions, needs, challenges, and desiderata. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2).

Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.

Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020). Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. CHI '20. https://doi.org/10.1145/3313831.3376445

Mitchell, M., et al. (2019). Model cards for model reporting. FAT '19.* https://doi.org/10.1145/3287560.3287596

Pineau, J., et al. (2021). Improving reproducibility in machine learning research. JMLR, 22. https://www.jmlr.org/papers/v22/20-303.html

Raji, I. D., et al. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. FAT '20.* https://doi.org/10.1145/3351095.3372873

Refsgaard, J. C., & Henriksen, H. J. (2004). Modelling guidelines — terminology and guiding principles. Advances in Water Resources, 27(1), 71–82.

Sambasivan, N., et al. (2021). "Everyone wants to do the model work, not the data work": Data cascades in high-stakes AI. CHI '21. https://doi.org/10.1145/3411764.3445518

Sculley, D., et al. (2015). Hidden technical debt in machine learning systems. NIPS 2015.

Studer, S., et al. (2021). Towards CRISP-ML(Q). Machine Learning and Knowledge Extraction, 3(2), 392–413.

Voinov, A., & Bousquet, F. (2010). Modelling with stakeholders. Environmental Modelling & Software, 25(11).

Voinov, A., et al. (2018). Tools and methods in participatory modeling. Environmental Modelling & Software, 109. https://doi.org/10.1016/j.envsoft.2018.08.028

Full citation list: references/citations.md.