Earn the data

Earn the Data

The principle is in the name: anyone can find data; this skill is about earning the right to use it. A dataset that loosely matches a topic is not the same as a dataset that can defensibly support a project's claims. Every recommendation produced by this skill must be defensible at the primary source, against the project's decision unit, on the rubric, and with named biases.

This is a master's-research-grade discipline. Skipping steps produces work that looks rigorous but cannot survive scrutiny.

The anti-pattern this skill exists to prevent

Grab a CSV that loosely matches the topic → load it into a notebook → produce a chart → present.

This fails because nobody asked: where does the data come from, what does it actually measure, at what resolution, with what biases, and does it still answer the project's actual question? Every step below exists to block that anti-pattern.

When invoked, run this pipeline end-to-end

The pipeline is a single pass. Do not stop partway. Do not skip steps because the user "just wants a quick recommendation" — quick recommendations are the failure mode the skill exists to prevent.

Step 1 — Identify the decision unit

Read the project's research question and identify the smallest spatial, temporal, and spectral unit at which claims must hold. This is the constraint everything else must satisfy.

Examples:

• "Which Barcelona blocks are most heat-stressed?" → spatial unit: city block (~100 m); temporal: daily / seasonal; spectral: thermal IR
• "How does air quality vary by hour across Madrid neighborhoods?" → spatial: neighborhood (~500 m); temporal: hourly; spectral: gas concentration

Write the decision unit explicitly in the inventory. Every subsequent dataset is evaluated against it.

Step 2 — Hunt candidates across the six categories

A complete inventory considers all six. For each category that's relevant, list 2–3 plausible candidates so the rubric has comparisons to make.

The six categories:

1. Remote sensing — optical (Sentinel-2, Landsat 8/9, PlanetScope)
2. Remote sensing — thermal / radar (Landsat thermal, MODIS, Sentinel-1)
3. Climate / atmosphere reanalysis (ERA5, ERA5-Land, CHIRPS)
4. In-situ sensors (OpenAQ, Purple Air, EEA, national networks)
5. Biodiversity and ecosystems (GBIF, iNaturalist, regional atlases)
6. Built environment / socio-spatial (OpenStreetMap, municipal open data, CityGML, cadastral data)

See references/categories.md for primary-source URLs and category-specific notes.

If the project is in Barcelona / Catalonia, also load references/barcelona.md for the Open Data BCN catalogue, UTM31 default CRS, Spanish cadastral access, and Mediterranean-specific caveats.

Step 3 — Verify each candidate at the primary source

Do not trust secondhand summaries — including this skill's own reference files. For each candidate:

• Fetch the primary source (the dataset portal, the publication describing it, or the issuing authority's documentation page).
• Confirm it still exists and is currently maintained. Many datasets are deprecated, replaced by newer versions, or hosted by defunct institutions. Versions matter — FungalRoot v1 vs v2.0, ERA5 vs ERA5-Land, Sentinel-2 L1C vs L2A.
• Record the actual resolution, coverage, temporal range, and access method as you saw them at the source. Numbers carried over from memory or from the user's brief have a habit of being wrong.

If the primary source is unreachable, mark the candidate as unverified and surface this in the inventory — do not silently fall back to the secondhand number.

Step 4 — Apply the 2× resolution rule

For each verified candidate, compare resolution to the decision unit on every relevant axis (spatial, temporal, spectral).

The rule: data resolution should be at least 2× finer than the decision unit on each axis where claims will be made. Anything coarser cannot defensibly support claims at that scale, no matter how clever the analysis.

Concrete failure example: ERA5's ~30 km grid means central Barcelona spans roughly 1.5 cells. Neighborhood-level claims from ERA5 are not supportable, full stop. Sentinel-2 at 10 m allows block-level claims for the same city.

Drop candidates that fail the 2× rule for the project's main claim. They may still appear in the inventory as supporting context, but flag clearly that they cannot anchor the analysis.

Step 5 — Score surviving candidates on the 5-dimension rubric

Score each on a 0–3 scale across all five dimensions: Provenance, Resolution, Coverage, Licensing, Bias.

Always show all five scores, not just the total. The shape of the score matters more than the sum — a 12/15 with Bias = 0 is far weaker than a 12/15 with Bias = 2, and the user needs to see that to defend the choice.

See references/rubric.md for the full scoring guide and worked examples.

Step 6 — Draft a data sheet for each recommended candidate

The data sheet is the artifact that earns the right to use the data. Use the 8-section template (the seven from Gebru et al. 2021, plus a Limitations section added for this discipline):

1. Motivation
2. Composition
3. Collection
4. Pre-processing
5. Uses (most critical — what the data should and should not be used for)
6. Distribution
7. Maintenance
8. Limitations

Open data does not mean no responsibility. Sections 5 and 8 are where defensibility lives — most data misuse comes from skipping them.

See references/data-sheet-template.md for the fillable template.

Step 7 — Flag coordinate-system and unit pitfalls

For each recommended dataset, name explicitly:

• Coordinate system — lat/lon (degrees, cannot compute area or distance directly), UTM zones (projected, allows area / distance / buffers — UTM31 for Barcelona, UTM30 for Madrid, etc.), or local national grids
• Unit conversions required — Kelvin → Celsius for ERA5 air temperature; J/m² → W/m² for radiation; UTC → local time only at the presentation layer; DN → reflectance for satellite imagery
• Reprojection requirements when joining datasets — never join two layers in different coordinate systems without reprojecting first

These are the silent-failure category — analyses that run without errors but produce nonsense. Each goes in the data sheet's Pre-processing section as a step the user must perform before use.

See references/pitfalls.md for the common-mistake catalogue.

Step 8 — Name sampling biases as first-class findings

For each recommended dataset, name at least one specific bias and how it would propagate into this project's analysis. Do not write "biases may exist" — write the specific mechanism. Specificity is the whole point.

Useful level of specificity:

• Purple Air sensors are oversampled in wealthy neighborhoods with single-family housing; air-quality maps will show artificially good readings in low-density areas and absent data in dense low-income zones.
• iNaturalist research-grade fungi records are dominated by visible fruiting bodies; arbuscular mycorrhizal fungi, which do not fruit visibly, are essentially invisible to the dataset.
• OpenStreetMap building footprints are densest where mappers live; rural and recently-developed areas can be 30–60% incomplete.
• GBIF occurrences over-represent areas near research institutions and tourist routes.

If a bias is not characterizable, that is itself a finding — flag it and recommend cross-validation against an independent source.

Step 9 — Generate the profiling plan

Profiling is what turns "data the user has" into "data the user understands." The user must run this themselves before committing to use the data. The skill produces the plan; the user produces the verification.

The eight essential profiling cells:

1. Shape and dtypes — rows, columns, types per field
2. Missing values — count and pattern per field; are missings random or structured?
3. Numeric summaries — min, max, mean, median, distribution shape, outliers
4. Categorical summaries — unique values, value counts, taxonomy consistency
5. Spatial coverage — bounding box, point density per zone, holes
6. Temporal coverage — date range, gaps, sampling cadence
7. Cross-field consistency — do related fields agree (e.g., reported lat/lon inside reported region)?
8. Bridging methods — what fields, keys, or joins connect this dataset to others in the inventory?

Output the profiling plan listing these eight checks for each recommended dataset, with notes on what specifically to look for in this project's context.

Step 10 — Revisit the original brief

The final and most important step. Given what the data actually is — its resolution, coverage, biases, limitations — ask:

Does the original research question still hold?

Three possible answers:

• Yes, unchanged — proceed.
• Yes, but narrower — the question survives if scoped to the resolution / coverage the data actually supports. Specify exactly how the question must be narrowed.
• No — the question cannot be answered with available data at the required scale. Specify what the data can answer, and surface this as a finding the user must take back to the brief.

This step is non-negotiable. A project that pushes forward with data that cannot support the question is the failure mode the rubric exists to prevent.

Output artifacts

Write all four to disk in a phase-2/ directory at the project root (or wherever the user invoked the skill):

• data-inventory.md — every candidate considered, with rubric scores and reasoning for inclusion / exclusion. Decision unit at the top.
• data-sheets/<source>.md — one 8-section data sheet per recommended source.
• profiling-plan.md — the 8-cell profiling checklist for each recommended dataset, with project-specific notes.
• brief-revisit.md — the answer to Step 10 with concrete recommendations.

Then summarize the final recommendations inline, with rubric scores and the brief-revisit conclusion as the headline.

Out of scope

• Sketches, decision maps, output mockups — different artifacts, different skill.
• Running profiling notebooks — the user runs those on their own data, after the skill produces the plan.
• Recommendations from a single source without triangulation — primary validated by secondary, always.
• Skipping the brief-revisit because the data looks promising — even good data may not answer the question that was asked.

Why the discipline matters

Data resolution determines whether meaningful analysis is possible. No amount of code quality compensates for a 30 km grid trying to answer a 100 m question. The architect's anti-pattern (grab → visualize → present) produces deliverables that look professional and fail the moment a stakeholder asks "where does this come from?"

The skill enforces a different sequence: justify, then use. The artifacts are the justification. The user defends them.