Review data pipeline

Empirical Research Pipeline Review

Purpose

Audit experiment specs and empirical pipelines for claim-threatening validity and reproducibility failures.

When To Use

Use before launching a spend-sensitive or time-sensitive experiment, when an empirical workflow may contain silent failure modes, or when a project needs an evidence register for data, model, analysis, or evaluation claims.

This skill owns experiment-spec review, pipeline audit, evidence-register production, and pipeline-specific reproducibility checks. update-docs owns dated documentation updates after accepted findings exist.

Required Inputs

• {{PROJECT_NAME}}
• {{TARGET_PIPELINE_OR_ARTIFACT}}
• {{PROJECT_CONTEXT}}
• {{ARTIFACT_LOCATIONS}}
• {{CANONICAL_CONTEXT_ROOT}}
• {{AS_OF_DATE}}

Optional Inputs

• {{EXPERIMENT_SPEC}}
• {{EXISTING_REPORTS_AND_NOTES}}
• {{AUDIT_FINDINGS}}
• {{SHARED_RESEARCH_CONTRACT}}
• {{RISK_CONTEXT}}
• {{STYLE_AND_LINKING_CONVENTIONS}}
• {{AVAILABLE_TOOLS}}
• {{OUTPUT_LOCATION}}

Authority Order

Use {{SHARED_RESEARCH_CONTRACT}} when supplied. Otherwise apply this order by claim type:

1. Explicit user instructions, release constraints, safety constraints, and local project conventions.
2. Raw or near-raw project evidence for pipeline claims.
3. Primary external sources for external claims.
4. Reproducible analyses and transparent secondary sources that cite primary artifacts precisely.
5. Canonical current-state project documents and authority ledgers.
6. Dated archival records and historical notes.
7. Input reports, model outputs, comments, and prior narratives, treated as leads.
8. Auditor judgment, labeled as judgment.

Raw pipeline evidence includes data, logs, configs, code, run metadata, notebooks, dashboards, analysis outputs, and source artifacts.

Tool And File-Access Assumptions

Use repository search, file inspection, code execution, notebooks, dashboards, logs, and document editing when available and safe. Commands that are spend-sensitive, privacy-sensitive, destructive, or irreversible require an explicit approval gate from {{PROJECT_CONTEXT}}.

If commands cannot be run, produce a manual check plan with expected evidence and failure modes. If artifact creation is unavailable, output the complete review in Markdown.

Handoffs

Upstream artifacts are experiment specs, pipeline code, datasets, manifests, configs, logs, notebooks, dashboards, existing reports, and canonical context.

Downstream artifacts are Experiment Spec Review, Review Plan, Evidence Register, Pipeline Audit, and Documentation Handoff Plan.

Pass accepted empirical findings to literature-review when they affect a project contribution, to research-manuscript-audit when they constrain a draft claim, or to update-docs when current documentation needs dated supersession.

Claim-Support Statuses

Use one status per nontrivial finding:

• Supported
• PartiallySupported
• ReasonableInference
• Ambiguous
• Overstated
• Unsupported
• StaleOrSuperseded
• Unverified
• Opinion

Process

Choose the smallest pass that answers the user request.

Experiment Spec Review

Use before execution when {{EXPERIMENT_SPEC}} is supplied.

Check:

• research question and claim under test
• datasets, manifests, splits, leakage risks, and contamination risks
• model, tokenizer, dependencies, hardware assumptions, and external services
• code entry points and required existing modules
• parameters, grids, seeds, checkpoints, output artifacts, and expected schemas
• success criteria, stopping criteria, uncertainty handling, and baseline requirements
• gates, canaries, untouched surfaces, matched controls, transfer checks, and externality checks
• dry-run plan and silent-failure tests
• spend, privacy, destructive-action, release, and human-approval thresholds
• resume plan, monitoring plan, and provenance fields

Review Plan

Use when scope, assumptions, risks, or evidence requirements are unclear.

Plan:

• claims under review
• artifact inventory
• provenance gaps
• evaluation design
• risks and claim-linked failure modes
• evidence needed to update confidence
• blockers and prerequisites

Pipeline Evidence Register

Use after a review plan exists or when the target is well-defined.

Inspect:

• data lineage, collection, labeling, and target construction
• preprocessing, filtering, joins, feature engineering, and caching
• split logic, contamination, duplication, and leakage
• sampling bias, imbalance, missingness, and slice coverage
• metric computation, aggregation, baselines, controls, and uncertainty
• seed sensitivity, instability, and reproducibility
• configuration drift, schema drift, dependency drift, and external services
• manual steps and operational fragility
• privacy, security, misuse, fairness, monitoring, or release risks where relevant

For every nontrivial finding, record finding ID, claim under review, support status, evidence type, evidence summary, counterevidence, interpretation, affected metric or control, impact, confidence, flags, and follow-up.

One-Pass Pipeline Audit

Use when the pipeline is small enough for one pass or when a complete review package is needed quickly. Run the review plan, evidence register, and final report in one pass.

Documentation Handoff Plan

Use after audit findings are accepted and docs may need updates. Produce a dated handoff for update-docs with affected files or sections, claims to update, evidence links, severity, and recommended supersession notes. This pass produces a handoff package; update-docs applies documentation changes.

Output Contract

Produce the applicable Markdown artifact.

Experiment Spec Review

1. Verdict: Launchable, LaunchableWithCaveats, or DoNotLaunch.
2. Blocking Issues Before Launch
3. Major Issues
4. Minor Issues
5. Claim-Linked Control Gaps
6. Provenance And Resume Gaps
7. Human Approvals Required
8. Minimum Patch Set

Review Plan

1. Project Summary
2. Claims Under Review
3. Scope, Assumptions, And Constraints
4. Stakeholders Or Affected Populations
5. Artifact Inventory
6. Provenance And Reproducibility Gaps
7. Evaluation Design
8. Risks And Claim-Linked Failure Modes
9. Evidence Needed To Update Confidence
10. Blocking Gaps And Prerequisites

Evidence Register Or One-Pass Audit

1. Scope And Claims
2. Artifact Inventory And Provenance Gaps
3. Evaluation Plan And Acceptance Thresholds
4. Evidence Register
5. Supported Findings
6. PartiallySupported And ReasonableInference Findings
7. Ambiguous, Unsupported, StaleOrSuperseded, And Unverified Findings
8. Risks, Limitations, And Uncertainty Register
9. Recommended Fixes Or Follow-Up Tests
10. Documentation Handoff Plan

Documentation Handoff Plan

1. Accepted Findings
2. Claims To Supersede
3. Affected Files Or Sections
4. Evidence Links
5. Recommended update-docs Inputs
6. Remaining Disagreements Or Missing Evidence

Write to {{OUTPUT_LOCATION}} when supplied and artifact creation is available.

Quality Gate

Before finalizing, verify:

• The review decides whether the run or pipeline can answer the target question.
• Every blocker is tied to a claim, cost, safety, privacy, or reproducibility risk.
• Silent-failure risks are surfaced for the central claim.
• Evidence and interpretation are separated.
• Missing metadata or results are marked missing.
• Weak or indirect evidence supports narrow claims.
• Commands, checks, edits, and artifact claims match completed actions.
• Documentation changes are handed to update-docs as dated evidence.