Crispdm phase 2 companion

CRISP-DM Phase 2 Companion

A thin companion to earn-the-data. Where earn-the-data enforces a ten-step discipline for choosing datasets, this skill enforces what the canonical CRISP-DM reference model (Chapman et al., 2000) and the post-2018 dataset-documentation literature demand for closing out Phase 2 — the reproducibility, validation, and machine-readable-metadata artifacts a downstream architect (or auditor, or future-you) needs in order to trust the work.

This skill does not re-do anything earn-the-data already did. It rides on top of earn-the-data's outputs and adds the eight artifacts the literature names as load-bearing.

1. What earn-the-data covers vs. what this skill adds

Concern	earn-the-data	crispdm-phase-2-companion
Decision unit	YES (Step 1)	reads it
Six-category candidate hunt	YES (Step 2)	reads inventory
Primary-source verification	YES (Step 3)	reads inventory
2× resolution rule	YES (Step 4)	reads inventory
5-dimension rubric (Provenance, Resolution, Coverage, Licensing, Bias)	YES (Step 5)	reads scores
8-section data sheets	YES (Step 6)	reads, then exports to Croissant
CRS / unit pitfalls	YES (Step 7, narrative)	extends to MAUP + edge effects + binding declaration
Sampling biases	YES (Step 8, narrative)	extends to label/annotation-noise estimate (separated)
Profiling plan (human-runnable EDA)	YES (Step 9)	converts to executable schemas
Brief-revisit	YES (Step 10)	reads, never overwrites
Ingestion log (timestamps, SHA-256, agent, errors)	NO	YES (Step B) — closes G1
Ingested-data description (observed, not asserted)	NO	YES (Step C) — closes G2
Wang & Strong / ISO 25012 cross-check	NO	YES (Step D) — closes G3
Executable schema (TFDV / Pandera / GE / frictionless)	NO	YES (Step E) — closes G4
MAUP and edge-effect declaration	implicit	YES (Step F) — closes G5
Annotation/label-noise estimate	subsumed under bias	YES (Step G) — closes G6
Croissant JSON-LD sidecar	NO	YES (Step H) — closes G7
Versioning policy	NO	YES (Step I) — closes G8
Phase 3 handoff gate	NO	YES (Step J)

Used separately, each skill is incomplete. Used together, they cover Phase 2 to the standard the academic literature now expects.

2. Hard precondition — refuse to start if these are missing

This skill is a companion. It will not run if earn-the-data has not run. Before doing anything, check that all of the following exist in the project's phase-2/ directory (or wherever earn-the-data wrote its outputs):

phase-2/
├── data-inventory.md          # earn-the-data Step 5 — rubric scores
├── data-sheets/               # earn-the-data Step 6 — one .md per recommended source
│   └── <source>.md
├── profiling-plan.md          # earn-the-data Step 9 — 8-cell EDA checklist
└── brief-revisit.md           # earn-the-data Step 10 — does the question still hold

If any of these are missing, halt and instruct the user:

This skill is a Phase 2 close-out companion. It requires the outputs of earn-the-data. Missing: <list of missing files>. Run earn-the-data first, then re-invoke this skill.

Do not attempt to reconstruct, guess, or paper over missing inputs. The halt is correct behavior.

3. Required inputs

In addition to the four earn-the-data artifacts above, this skill needs read-access to:

• The raw retrieved files themselves (so SHA-256 hashes, file sizes, row/column counts, and observed value ranges are observed at the bytes, not asserted from memory or from the brief).
• The ingestion script(s) or notebook(s) used during Phase 2 retrieval (so the agent field of the ingestion log captures the actual script name, git SHA, and runtime version — not a guess).
• The project root path, so artifacts are written next to the earn-the-data outputs (typically phase-2/).

If retrievals were ad-hoc (manual clicks on a portal, unscripted downloads), reconstruct what can be reconstructed and flag the rest with unreproducible: yes plus a remediation note. Do not silently fabricate a reproducible-looking log over an unreproducible retrieval.

4. Specialist lenses

Codex runs single-agent — there is no sub-agent spawning here. Where the original workflow routed a step to a specialist, adopt that specialist's lens inline: reason through the step as that expert would before writing the artifact. For an independent second opinion (e.g. Phase 5 verification), run the step in a fresh Codex session primed with the specialist role instead.

Curated for Phase 2 companion:

• Data Engineer — schema validation, executable profiling (TFDV/Pandera/Great Expectations/frictionless), Croissant sidecar generation
• Database Optimizer — schema-level checks on ingested tabular data
• AI Data Remediation Engineer — anomaly detection in ingested batches

5. Sequential steps (A–J)

Run as a single pass over the recommended datasets in data-inventory.md. Do not parallelize across datasets unless re-runs are independently safe. Each step writes a named file into phase-2/.

Step A — Verify earn-the-data outputs exist

Walk the precondition checklist (Section 2). If any of data-inventory.md, data-sheets/, profiling-plan.md, or brief-revisit.md is missing, halt with the remediation message in Section 2. This is the only halt condition that is normal — every later step that fails should surface the failure but not halt the whole pass.

Step B — Build the Ingestion Log → phase-2/ingestion-log.md

For each retrieval performed during Phase 2, record one entry with:

• retrieval_timestamp_utc — ISO 8601, UTC, second-precision
• source_name — matches a row in data-inventory.md
• url_or_api_call — exact URL with parameters, or the API call expression
• agent — script filename + git SHA (or manual if ad-hoc)
• runtime — language + version (e.g., python 3.11.6)
• destination_path — relative path under project root
• sha256 — SHA-256 of bytes as downloaded, before any decoding
• size_bytes — integer
• content_type — observed HTTP content-type or detected MIME
• encoding — observed (utf-8, latin-1, binary, etc.)
• http_status — HTTP code or API status
• errors — any errors encountered (empty string if none)
• fallback — fallback adopted if the primary path failed (empty string if none)
• reproducible — yes / no with a one-line reason if no

Use the template in references/ingestion-log-template.md. Closes G1 (Chapman et al., 2000 — Initial Data Collection Report).

Step C — Build the Ingested-Data Description → phase-2/ingested-data-description.md

For each recommended source, record observed (not asserted) values:

• row count (or feature count for vector spatial data; pixel count or band count for raster)
• column list with inferred dtype (per-column)
• observed min / max for numeric columns
• observed cardinality and top-10 values for categorical columns
• file format and on-disk size in bytes
• detected encoding
• the date of observation

Distinguish this artifact from the data sheet explicitly. The data sheet describes what the data is in general (assertional, drawn from documentation). The ingested-data description describes what was actually pulled into this project (observational, drawn from the bytes on disk). When the two disagree, that disagreement is a finding — surface it in quality-cross-check.md under Currentness or Credibility.

Closes G2.

Step D — Wang & Strong / ISO 25012 cross-check → phase-2/quality-cross-check.md

Build a matrix: one row per recommended source × one column per quality dimension. Use one consistent dimension set — either Wang & Strong's 15 dimensions (four categories) or ISO/IEC 25012's 15 characteristics (inherent + system-dependent). See references/dq-dimensions.md for both.

For each cell, record one of:

• assessed: <one-sentence summary>
• deferred: <reason — typically scope, time, or unavailable evidence>
• n/a: <reason>

Do not record 0/3 or omit dimensions silently. Absence of evidence is not evidence of absence — the cross-check exists to make scope explicit.

The 5-dimension earn-the-data rubric (Provenance, Resolution, Coverage, Licensing, Bias) maps onto a subset of Wang & Strong. The cross-check makes explicit which dimensions are not covered by the rubric so the architect can defend the rubric's narrower scope.

Closes G3 (Wang & Strong, 1996; ISO/IEC 25012:2008).

Step E — Generate executable schemas → phase-2/schemas/<source>.yaml

For each recommended source, emit a schema file in one of:

• TensorFlow Data Validation Schema protobuf
• Great Expectations ExpectationSuite
• Pandera DataFrameSchema
• frictionless Table Schema

Pick the one already in use in the project. Default: Pandera for tabular Python projects, frictionless for general-purpose / language-agnostic pipelines. The schema must encode at minimum:

• field name and dtype
• nullability and missingness budget
• allowed-value set for low-cardinality categoricals
• min / max for numerics — drawn from the data sheet, not from the ingested bytes. Schemas are expectations, not observations. (Breck et al., 2019: inferred schemas are a starting point, not a finished artifact.)
• key / index / uniqueness constraints
• CRS expectation for spatial layers (EPSG code)

Then validate the schema against at least one row from the ingested file. A schema that does not parse the data it claims to describe is worse than no schema — surface the failure inline.

Closes G4 (Breck et al., 2019; Polyzotis et al., 2018).

Step F — Geospatial declarations → phase-2/geospatial-declarations.md

For any spatial dataset, declare:

• Native CRS at source (EPSG code, e.g., EPSG:4326 for GBIF point data)
• Chosen analysis CRS (EPSG code, e.g., EPSG:25831 for Barcelona UTM31N) and reprojection method (nearest / bilinear / cubic for raster; not-applicable for vector)
• Areal unit chosen for aggregation (cell size for grids; admin level for polygons)
• MAUP sensitivity note — was an alternative unit tested? If not, why not? (Openshaw & Taylor, 1979)
• Edge-buffer policy — how observations near the study-area boundary are handled (buffer extent in metres, or marker column on edge records)

See references/maup-and-edge-effects.md for context and worked examples. Closes G5.

Step G — Bias and annotation-quality estimate → phase-2/bias-and-annotation.md

Two parts.

Part 1. Lift the bias findings already named in earn-the-data Step 8 (typically in data-sheets/<source>.md or data-inventory.md) into this artifact verbatim, so the artifact is self-contained and reviewable without flipping between files.

Part 2. For each crowd-annotated or citizen-science source (iNaturalist, GBIF, OpenStreetMap, Purple Air, eBird, etc.), estimate the label/annotation error rate:

• cite any published audit if one exists (e.g., Northcutt et al., 2021, for ImageNet-style benchmarks)
• otherwise mark as unknown and recommend an independent cross-validation source

Northcutt et al. (2021) showed that label noise destabilizes downstream model rankings even at ~3% error rates. This artifact records what the project knows, what it does not know, and what it plans to do about both.

Closes G6.

Step H — Croissant sidecar → phase-2/croissant/<source>.jsonld

For each recommended source, emit a machine-readable Croissant JSON-LD document (Akhtar et al., 2024) populated from the data sheet's Motivation, Composition, Collection, Distribution, and Maintenance sections.

Use the MLCommons Croissant editor (https://mlcommons.github.io/croissant/) or the mlcroissant Python library. See references/croissant-sidecar-template.md for a valid JSON-LD example with @context, @type: sc:Dataset, recordSet, field, distribution, and license fields populated.

When the JSON-LD and the Markdown data sheet disagree, the data sheet wins — the JSON-LD is a downstream serialization, not the source of truth.

Even if the project never publishes, Croissant sidecars make the artifacts ingestible by Hugging Face, Kaggle, Google Dataset Search, and OpenML — the standardization bet most likely to outlive any single proprietary format.

Closes G7.

Step I — Versioning policy → phase-2/versioning-policy.md

Short statement covering:

• Pinning — how dataset versions are pinned (URL with version tag, snapshot hash, archived copy in DVC / Zenodo / S3)
• Snapshot location — where pinned snapshots live, with access instructions
• Re-ingest cadence — one-shot / monthly / on-demand
• Breaking-change detection — re-run the executable schema (Step E) and diff against the previous run; what counts as a breaking change vs. a tolerable change
• Retirement — when and how a dataset is removed from the inventory (e.g., upstream deprecation, license change, primary source disappearance)

Closes G8 (Hutchinson et al., 2021).

Step J — Phase 3 handoff manifest → phase-2/phase-3-handoff.md

One page listing every artifact in phase-2/ (earn-the-data outputs + companion outputs from this skill) with:

• a one-line description of what the artifact contains
• the gate test: "Phase 3 can begin if and only if this artifact exists and is non-empty"
• the owner (script, person, or team) responsible for the artifact

This is the gate. Phase 3 (Data Preparation) does not start until every row in the handoff manifest passes the gate test.

6. Exit criteria (the skill's own self-audit)

Before declaring done, the skill must confirm all of the following. If any fails, surface the failure inline and do not produce a final "ready for Phase 3" summary that papers over a failure.

• E1. Every recommended source in data-inventory.md appears in ingestion-log.md, ingested-data-description.md, schemas/, and croissant/. No silent skips.
• E2. Every SHA-256 in the ingestion log is computable from a file currently on disk, or marked reproducible: no with a one-line remediation note.
• E3. Every schema in schemas/ validates against at least one row from the corresponding ingested file. Schemas that don't parse the data they purport to describe are surfaced as failures.
• E4. Every spatial dataset in data-sheets/ appears in geospatial-declarations.md with both native CRS and analysis CRS declared.
• E5. Every crowd-annotated source (iNaturalist / GBIF / OpenStreetMap / Purple Air / eBird / similar) has an annotation-quality estimate or an explicit unknown with a planned mitigation.
• E6. phase-3-handoff.md lists every artifact and the gate condition for each.
• E7. Every recommended source has a Croissant sidecar (croissant/<source>.jsonld) that is valid JSON-LD and consistent with the Markdown data sheet.

7. Anti-patterns (what this skill must NOT do)

• Re-doing the rubric. The 5-dimension rubric is earn-the-data's domain. If the architect wants a different rubric (e.g., full Wang & Strong), produce the cross-check (Step D); do not re-score.
• Re-doing the brief-revisit. Step 10 of earn-the-data is the most important single output of Phase 2. This skill must not silently overwrite brief-revisit.md.
• Producing schemas from observed data without expert review. TFDV documentation is explicit (Breck et al., 2019): inferred schemas are a starting point, not a finished artifact. Schemas in this skill are seeded from the data sheet (expert-authored), not auto-inferred from ingested bytes.
• Treating absence of evidence as evidence of absence. If a quality dimension cannot be assessed, the cross-check records deferred with a reason. It does not score the dimension 0/3 or omit it silently.
• Producing Croissant sidecars that disagree with the data sheets. The Markdown data sheets are the source of truth; the JSON-LD is a downstream serialization. If the two disagree, the data sheet wins and the JSON-LD is regenerated.
• Running before earn-the-data. This is the only hard precondition. Halting is correct behavior.
• Papering over an unreproducible retrieval. If the original ingest was ad-hoc, mark it reproducible: no with a remediation note. Do not fabricate a clean log.

8. Handoff to Phase 3 (Data Preparation) — data-contract precursors

Phase 3 starts when phase-3-handoff.md exists, every artifact it lists exists and is non-empty, and the user signs off. The Phase 3 skill (when it exists) should consume the Phase 2 artifacts as data-contract precursors:

• schemas/<source>.yaml — the validation entry point for any prepared data. Phase 3 transformations must produce outputs that satisfy a derived schema, and the derivation must be traceable back to this one.
• geospatial-declarations.md — the binding declaration of analysis CRS and aggregation unit. Phase 3 must not silently reproject, re-grid, or re-aggregate; any change is a deliberate, recorded decision.
• versioning-policy.md — the rule for whether a re-ingest is required before preparation begins. If the policy says monthly and the last ingest was 45 days ago, re-ingest before continuing.
• brief-revisit.md — the binding scope statement. Any Phase 3 work outside the brief-revisit's surviving question must be flagged and re-discussed with the user. Phase 3 must not silently narrow the brief-revisit (silently choosing a subset of recommended datasets); narrowings must be explicit and recorded.
• quality-cross-check.md + bias-and-annotation.md — the known-unknowns register. Phase 3 mitigations (imputation, filtering, weighting) must be traceable to a row in one of these documents.
• croissant/<source>.jsonld — the machine-readable handshake. If Phase 3 produces a derived dataset for re-publication, its Croissant sidecar inherits provenance from these.

Together these six files form the contract that Phase 3 consumes. Without them, Phase 3 is operating on assumptions, not evidence — which is the data-cascade failure mode Sambasivan et al. (2021) document.

9. References

See references/citations.md for the full bibliography. The skill operationalizes:

• Chapman et al. (2000) — CRISP-DM 1.0 reference model; the four-task Phase 2 block
• Wang & Strong (1996) — 15-dimension data-quality framework (Step D)
• ISO/IEC 25012:2008 — alternative 15-characteristic data-quality model (Step D)
• Gebru et al. (2021) — Datasheets for Datasets; data-sheet template
• Pushkarna et al. (2022) — Data Cards; purposeful documentation
• Bender & Friedman (2018) — Data Statements for NLP; annotator-demographic documentation
• Holland et al. (2018) — Dataset Nutrition Labels; at-a-glance machine-generated summaries
• Hutchinson et al. (2021) — dataset accountability; versioning policy (Step I)
• Breck et al. (2019) — data validation for ML; executable schemas (Step E)
• Polyzotis et al. (2018) — data lifecycle in production ML; schema-as-code
• Northcutt et al. (2021) — pervasive label errors; label-noise estimate (Step G)
• Openshaw & Taylor (1979) — MAUP; geospatial declarations (Step F)
• Wilkinson et al. (2016) — FAIR principles; provenance / lineage
• Akhtar et al. (2024) — Croissant metadata format; JSON-LD sidecar (Step H)
• Sambasivan et al. (2021) — data cascades in high-stakes AI; the failure mode this skill exists to prevent
• Heger et al. (2022) — practitioner perceptions of data documentation
• Studer et al. (2021) — CRISP-ML(Q); quality-assurance extension to CRISP-DM
• Schröer et al. (2021) — systematic review of CRISP-DM applications
• Tukey (1977) — exploratory data analysis foundations

See also:

• references/gap-analysis.md — earn-the-data step × canonical Phase 2 task × gap
• references/ingestion-log-template.md — fillable template for Step B
• references/croissant-sidecar-template.md — valid JSON-LD example for Step H
• references/dq-dimensions.md — Wang & Strong 15 + ISO 25012 15 (Step D)
• references/maup-and-edge-effects.md — context and worked examples (Step F)