Crispdm phase 6 deployment

CRISP-DM Phase 6 — Deployment (Pipeline Handoff)

The deliverable is a reproducible pipeline + a documented data product + a maintenance plan, all handed off to a named user. It is not a dashboard, not a web app, not an API, not a hosted service.

This skill enforces that reframing.

---

1. Reframing Phase 6 for pipeline-only projects

Chapman et al. (2000) define Phase 6 around four generic tasks: Plan Deployment, Plan Monitoring and Maintenance, Produce Final Report, Review Project. In industrial readings (Sculley et al., 2015; Paleyes et al., 2022), "deployment" almost always means operationalizing a trained model behind an API. That framing does not fit a pipeline-only graduate seminar.

Under the pipeline-only reframing, the four canonical tasks become:

Canonical task	Pipeline-handoff reading
Plan Deployment	Plan release packaging and publication of pipeline + data product
Plan Monitoring and Maintenance	Plan upstream-source monitoring, dependency refresh, ownership transfer
Produce Final Report	Produce final report + datasheets + intended-use statement + methods document
Review Project	Open retrospective with lessons-learned register and CRediT contributor accounting

Phase 6 here is the act of closing the loop: making the work citable, auditable, re-runnable, and extensible by someone other than its author.

---

2. The anti-pattern this skill exists to prevent

Zip up the notebooks → email them to the user → call it "deployed."

This fails because nobody else can re-run them. No lockfile. No container. No datasheet. No DOI. No named maintainer. The pipeline is born dead the moment the author moves on. Every step below blocks that anti-pattern.

A close cousin of the anti-pattern is building a dashboard "because the user needs to see something." The course constraint says no. If the user later needs a UI, that is a separate downstream project. Stay in your lane: be the architect of the data system, not the front-end engineer.

---

3. Hard precondition — refuse to run without these

The skill blocks until all six inputs are present:

1. Phase 5 go/no-go memo — an explicit decision document. If decision: no-go, stop and route back to Phase 1. If missing, stop and report.
2. Pipeline notebooks/scripts — version-controlled, ordered, runnable end-to-end.
3. Data product — the actual output artifact(s) (CSV / GeoPackage / GraphML / Parquet) sitting in outputs/.
4. Phase 4 modeling / feature notes — the design decisions to carry into the final report.
5. Intended user named — a real role at a real organization, even if hypothetical (e.g., "capital planning analyst at Ajuntament Espais Verds Barcelona").
6. Decision log — every defensible design choice made earlier in the project, in any format.

If any input is missing, name it in a numbered checklist and stop. Do not fake-run Phase 6 on a half-built pipeline.

---

4. Required inputs (recap and location)

Input	Expected location	Format
Phase 5 go memo	research/crispdm/05-evaluation-go-memo.md (or similar)	Markdown, decision: go line
Pipeline code	notebooks/, src/, Snakefile	Versioned, executable
Data product	outputs/*.gpkg, outputs/*.csv, outputs/*.graphml	One or more artifacts
Phase 4 notes	research/crispdm/04-modeling-*.md	Markdown
Intended user	release/intended_user.md (creates this)	Name + role + decision context
Decision log	release/decision_log.md or sibling location	Markdown

---

5. Specialist lenses

Codex runs single-agent — there is no sub-agent spawning here. Where the original workflow routed a step to a specialist, adopt that specialist's lens inline: reason through the step as that expert would before writing the artifact. For an independent second opinion (e.g. Phase 5 verification), run the step in a fresh Codex session primed with the specialist role instead.

Curated for Phase 6:

• DevOps Automator — lockfiles, containerization, CI/CD for re-runs, infra-as-code
• Technical Writer — final report, datasheets, intended-use statements, re-run guides
• Codebase Onboarding Engineer — "how-to-extend" guide writing, stranger-can-onboard test
• Developer Advocate — open-science release packaging, community handoff
• Compliance Auditor — license + FAIR audit if regulatory / funding constraints apply

---

6. The 12 sequential steps

The pipeline is a single pass. Do not stop partway. Do not skip steps because the user "just wants a quick release" — quick releases are the failure mode the skill exists to prevent.

Step 1 — Confirm Phase 5 go decision and load inputs

• Read the Phase 5 memo. Confirm decision: go.
• Inventory the six required inputs. List anything missing in a numbered checklist; if any required input is missing, stop and report.
• Identify the named intended user. Write their name + role + decision context to release/intended_user.md.

Step 2 — Pin the environment (lockfile + container)

Create or verify:

• pyproject.toml (preferred) or environment.yml.
• A lockfile: poetry.lock, conda-lock output, or a pip-tools compiled requirements.txt with hashes. A requirements.txt with version ranges is not sufficient.
• A Dockerfile (or apptainer.def) that builds a working environment from the lockfile. Pin the base image to a specific tag (no :latest).
• A .python-version (or equivalent) that pins the interpreter.
• Verify: docker build . succeeds; a smoke notebook executes inside the container.

See references/reproducibility-infrastructure.md for tool selection.

Step 3 — Stamp versions on every input and output

For every input dataset, record in release/manifest.json:

{
  "name": "Ajuntament BCN tree inventory",
  "version_or_date": "snapshot 2026-04-15",
  "url_or_doi": "https://opendata-ajuntament.barcelona.cat/...",
  "license": "CC-BY-4.0",
  "sha256": "…",
  "bytes": 0
}

For every output, assign a semantic version (MAJOR.MINOR.PATCH) and embed it in the filename:

• outputs/barrier_composite_v1.0.0.gpkg
• outputs/mycorrhizal_network_v1.0.0.graphml
• outputs/top15_priority_zones_v1.0.0.csv

Record SHA-256 of each output in release/manifest.json. If an input is a live API, record retrieval timestamp + exact query parameters.

Step 4 — Write a datasheet for every produced dataset

For each output, create outputs/datasheet_<name>.md following Gebru et al. (2021). The eight required sections are documented in references/final-report-template.md and the FAIR doc.

A README that says "here is the file" is not a datasheet. Every produced dataset has its own datasheet.

Step 5 — Run the FAIR compliance checklist

Walk through Wilkinson et al. (2016). For each item, record pass | fail | n/a + justification in release/fair_checklist.md. The full checklist (16 items across F, A, I, R) is in references/fair-checklist.md.

Any fail blocks release. Any n/a requires a written justification.

Step 6 — Choose publication channels deliberately

For each released artifact, decide and record in release/publication_plan.md:

• Citable archive (always): Zenodo DOI via GitHub release integration (for code), direct upload (for data).
• Domain repository (where applicable): GBIF, PANGAEA, ICPSR, EBI, etc.
• Geospatial portal (where applicable): OGC API Features, INSPIRE-compliant catalog.
• Consumer-facing portal (where applicable): city / regional / national open-data portal.
• Institutional repository (fallback / preservation): university dataverse, library CRIS.

See references/publishing-channels.md for selection criteria.

Step 7 — Pick licenses deliberately (code, data, docs are separate)

In release/licenses.md, record and justify three choices:

• Code license: MIT / Apache-2.0 / GPL-3.0.
• Data license: CC-BY-4.0 / ODbL / CC0.
• Documentation license: CC-BY-4.0 default.

Add LICENSE (code), LICENSE-DATA (data), and LICENSE-DOCS (docs) to the repo root. See references/licensing-guide.md — a README without a license is legally "all rights reserved," which means not open even if the author intended it to be.

Step 8 — Write the final report

Generate release/final_report.md (render to PDF via Quarto or Pandoc). The full template — including academic conventions (Code & Data Availability statement, Funding, CoI, AI/tool disclosure, CRediT roles) — is in references/final-report-template.md.

Step 9 — Write the supporting documents

• release/intended_use.md — who, what decisions it informs, what it must NOT be used for, failure modes.
• release/limitations.md — known biases, edge cases, scope boundaries.
• release/how_to_rerun.md — one page; docker build . && docker run … snakemake all.
• release/how_to_extend.md — the soft parts of the pipeline (which input to swap, which weight to change).
• release/decision_log.md — every defensible design choice and its rationale.
• CITATION.cff in repo root (Citation File Format v1.2).
• CONTRIBUTORS.md with CRediT roles signed by all contributors.

Step 10 — Write the monitoring and maintenance plan

In release/monitoring_plan.md, name (template in references/monitoring-plan-template.md):

• Inputs monitored (explicit list with check cadence).
• Dependencies monitored (lockfile rebuild test, link check).
• Triggers for a new release (upstream-source bumps, scientific revisions, user-reported failures).
• Named maintainer (or an explicit "unmaintained — fork freely" statement; ambiguity is not acceptable).
• Ownership transfer plan (what happens if the maintainer leaves the institution).
• Deprecation policy (when products are marked superseded; tombstone metadata preserved).
• Re-run cadence (annual / quarterly / on-demand).

Step 11 — Tag, archive, mint DOIs

• Tag the Git release: v1.0.0.
• Trigger Zenodo via GitHub integration → mint DOI for code.
• Upload data bundle to Zenodo → mint DOI for data.
• Update final_report.md with the minted DOIs.
• Update CITATION.cff with the code DOI.
• (Optional) Submit to Software Heritage for source-code archival.

Step 12 — Project retrospective

Add release/retrospective.md covering:

• What worked / what didn't / what next (three-column).
• Lessons-learned register (numbered, phase-tagged, severity-tagged).
• Reverse premortem: which risks materialized, which were avoided.
• Contribution accounting in CRediT roles, signed by all contributors.
• End-of-cycle triggers identified: what would start a new CRISP-DM iteration?

---

7. The 11 deliverables (release bundle manifest)

A complete Phase 6 produces:

1. Release bundle manifest — release/manifest.json (every input + every output with version, DOI, license, SHA-256).
2. FAIR compliance checklist — release/fair_checklist.md (all items pass or justified n/a).
3. Final report — release/final_report.md + rendered PDF.
4. Datasheets for every produced dataset — outputs/datasheet_*.md.
5. Monitoring and maintenance plan — release/monitoring_plan.md.
6. Re-run + extend guides — release/how_to_rerun.md, release/how_to_extend.md.
7. Decision log — release/decision_log.md.
8. Intended-use statement — release/intended_use.md.
9. Licenses — LICENSE, LICENSE-DATA, LICENSE-DOCS.
10. Citation files — CITATION.cff, CONTRIBUTORS.md.
11. Retrospective — release/retrospective.md.

Plus: DOIs — one for code, one for data, recorded in the final report.

---

8. The 5 quality checks (run before declaring Phase 6 complete)

8.1 FAIR check

Every item in release/fair_checklist.md is pass or n/a with justification.

8.2 Reproducibility check — the "stranger test"

Hand the release bundle to someone who has never seen the project. Ask them to follow how_to_rerun.md. If they cannot reproduce the published outputs without contacting the author, Phase 6 fails. Common failure points:

• Container won't build (un-pinned base image, missing system library).
• Notebooks have hidden state (cells run out of order).
• An input URL has rotted.
• A field in a datasheet is too vague to disambiguate values in the file.

8.3 Citability check

Search the project's title and the dataset's title. Both must surface a landing page with a DOI within the first page of results from a major search engine within a week of release.

8.4 Maintainability check

The repo has a named maintainer with a working contact. monitoring_plan.md names a successor or explicitly states "unmaintained — fork freely."

8.5 Stranger-can-extend test

A second stranger reads how_to_extend.md and identifies, without help, the three soft parts of the pipeline (which input to swap, which scoring weight to change, which sub-pipeline to add to).

---

9. The 11 anti-patterns (block release if observed)

1. requirements.txt with version ranges and no lockfile. Use a lockfile.
2. Notebooks that depend on cells executed out of order. Use "Restart and Run All."
3. Data in outputs/ without a datasheet. Every output gets a datasheet.
4. README without a license. "All rights reserved" is not open.
5. Same license for code and data. They have different needs; choose deliberately.
6. Input cited by bare URL only, no DOI, no snapshot date. Inputs vanish; record the snapshot.
7. "Maintainer: TBD." Name a maintainer or declare unmaintained — both acceptable; ambiguity is not.
8. A final report with no Code & Data Availability statement. Add it.
9. An intended-use statement that says "for research." Too vague. Name the decision the product is meant to inform.
10. Conflating data publication with software publication. Code and data are separate artifacts with separate licenses and separate DOIs. Publish both.
11. Building a frontend "because the user needs a dashboard." They don't. The course constraint is explicit: NO frontend. The deliverable is the data product, the documentation, and the reproducibility infrastructure. If a user genuinely needs a UI, that is a separate downstream project with its own CRISP-DM cycle. Do not slip a UI into Phase 6 of a pipeline-only project.

Full anti-pattern catalogue with rationale is in references/anti-patterns.md.

---

10. This is NOT building a frontend (explicit guidance)

The course explicitly bans frontend work in this project. This skill enforces that constraint. Specifically:

• No dashboards. No Streamlit, no Dash, no Power BI, no Tableau, no Looker.
• No web apps. No Flask, no FastAPI, no Next.js, no static site beyond the rendered final report.
• No interactive maps "for the user." Internal exploration notebooks are fine; a published Folium / Leaflet HTML is not the deliverable.
• No "lite" production model. No model serving, no scoring endpoint, no scheduled jobs in someone else's infrastructure.

What you do ship:

• The dataset (citable, FAIR, datasheeted).
• The pipeline (pinned, reproducible, containerized).
• The report (final report + methods + intended use + limitations).
• The plan (monitoring, maintenance, ownership, deprecation).

If the intended user later needs an interface, that is the start of a new CRISP-DM cycle — Phase 1 (Business Understanding) with the data product from this project as a documented input. The teacher's directive is unambiguous: stay in your lane — be the architect of the data system, not the front-end engineer.

This is repeated for emphasis because the failure mode is real: students with frontend skills routinely slip a "small UI" into Phase 6 and then defend it as "user-facing deployment." It is not. It is scope creep into a phase the course explicitly excluded. The skill blocks it.

---

11. Release bundle checklist (copy-pasteable)

Copy this into release/release_bundle_checklist.md and tick each item. Full checklist also in references/release-bundle-checklist.md.

RELEASE BUNDLE CHECKLIST — vMAJOR.MINOR.PATCH — YYYY-MM-DD

ENVIRONMENT
[ ] pyproject.toml or environment.yml present
[ ] Lockfile present (poetry.lock / conda-lock / pip-tools)
[ ] Dockerfile pins base image (no :latest) and builds clean
[ ] .python-version pins interpreter
[ ] Smoke notebook runs in the container

CODE
[ ] All notebooks executed top-to-bottom in a clean kernel; outputs committed
[ ] All scripts have docstrings and a __main__ guard where applicable
[ ] Pipeline DAG defined (Snakefile / Nextflow / CWL / hand-documented)
[ ] CITATION.cff at repo root
[ ] LICENSE (code) at repo root
[ ] CONTRIBUTORS.md with CRediT roles
[ ] README points to release/final_report.md and the DOIs

DATA
[ ] manifest.json lists every input with DOI, snapshot date, SHA-256, license
[ ] manifest.json lists every output with version, SHA-256, license
[ ] LICENSE-DATA at repo root (or per-dataset license headers)
[ ] Every output has a datasheet in outputs/datasheet_*.md
[ ] Coordinate reference systems explicitly stated for any spatial output
[ ] Field-level data dictionary for every output

DOCUMENTATION
[ ] release/final_report.md (with rendered PDF)
[ ] release/intended_use.md (who, what decisions, what NOT to use)
[ ] release/limitations.md
[ ] release/how_to_rerun.md (one page)
[ ] release/how_to_extend.md
[ ] release/decision_log.md
[ ] release/retrospective.md
[ ] LICENSE-DOCS at repo root

FAIR
[ ] release/fair_checklist.md complete; every item pass or justified n/a

PUBLICATION
[ ] release/publication_plan.md identifies all channels
[ ] Zenodo DOI minted for code (via GitHub release)
[ ] Zenodo DOI minted for data
[ ] DOIs recorded in final_report.md and CITATION.cff
[ ] Domain repository listing (if applicable)
[ ] Open-data portal listing (if applicable)

MAINTENANCE
[ ] release/monitoring_plan.md names maintainer
[ ] release/monitoring_plan.md sets re-run cadence
[ ] release/monitoring_plan.md sets deprecation policy
[ ] release/monitoring_plan.md sets ownership transfer plan

QUALITY GATES
[ ] FAIR check passed
[ ] Reproducibility ("stranger test") passed
[ ] Citability check passed
[ ] Maintainability check passed
[ ] Stranger-can-extend check passed

NEGATIVE CHECKS (must all be FALSE — i.e., none shipped)
[ ] No frontend / dashboard / web app shipped
[ ] No live model-serving endpoint
[ ] No bare URLs as data citations (DOIs only)
[ ] No "TBD" in maintainer field
[ ] No notebook with out-of-order cells

SIGN-OFF
Maintainer: ____________________
Date: ____________________
Release tag: ____________________

---

12. End-of-cycle triggers — what kicks off a new CRISP-DM iteration

CRISP-DM is explicitly iterative (Chapman et al., 2000, p. 13). Phase 6 is the closing and opening hinge of the loop. The following events return the project to Phase 1 (Business Understanding) with the previous deliverables as inputs:

1. A new business question raised by the user upon receiving the product (e.g., "Can we do this for nighttime cooling too?").
2. Upstream-source change invalidating an input (e.g., Urban Atlas 2024 supersedes the 2021 edition used in v1).
3. A scientific update to the underlying method (e.g., a new paper revises hyphal growth rates).
4. A user-reported failure mode (e.g., the index ranks an obvious false positive at the top, suggesting a missing variable).
5. A scheduled refresh (annual update for budget-cycle alignment).
6. A discovered bias that requires methodological change (e.g., the host-mismatch sub-score null over 95% of city).
7. A user request for a UI — this triggers a new CRISP-DM cycle for a separate frontend project; it does not retroactively expand Phase 6 of the current project.

When any of these fire, restart at Phase 1 with a revised problem statement. Cite the v1 release DOIs as inputs.

---

13. References

Full citations and standards list in references/citations.md. Headline sources:

• Chapman et al. (2000) — CRISP-DM 1.0: Step-by-step data mining guide.
• Wilkinson et al. (2016) — FAIR Guiding Principles. Scientific Data, 3, 160018.
• Gebru et al. (2021) — Datasheets for datasets. CACM, 64(12).
• Mitchell et al. (2019) — Model cards for model reporting. FAT* '19.
• Sculley et al. (2015) — Hidden technical debt in ML systems. NeurIPS 2015.
• Paleyes, Urma, & Lawrence (2022) — Challenges in deploying ML: A survey. ACM CSUR 55(6).
• Studer et al. (2021) — Towards CRISP-ML(Q). MAKE, 3(2), 392–413.
• Boettiger (2015) — Docker for reproducible research. ACM SIGOPS OSR.
• Peng (2011) — Reproducible research in computational science. Science, 334(6060).
• Brand et al. (2015) — CRediT taxonomy. Learned Publishing, 28(2).
• Klein et al. (2014) — Reference rot. PLOS ONE, 9(12).
• Mons et al. (2017) — Cloudy, increasingly FAIR. Information Services & Use, 37(1).

Standards: ISO 19115-1, W3C PROV-O, OGC API Features, Darwin Core (TDWG), GBIF, DataCite Metadata Schema 4.4, Citation File Format (CFF) v1.2, INSPIRE Directive 2007/2/EC.