Data pipeline design
Skill Amey-Thakur/AI-SKILLS/skills/data-engineering/data-pipeline-design
Plug-and-play skills and prompts for every AI coding agent
npx -y skills add Amey-Thakur/AI-SKILLS --skill data-pipeline-designAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Build data pipelines that rerun safely, backfill cleanly, and tolerate late data. Use when designing batch or streaming pipelines or fixing ones that need manual babysitting.
SKILL.md
3.0 KB, as published. Nobody here has run it
Data pipeline design
A pipeline is production software whose test is: can you rerun any day of it, twice, a year later, and get the same correct table? Everything here serves that rerun.
Method
- Make runs idempotent by partition overwrite. Each run owns a partition (usually a date/hour) and replaces it wholesale (INSERT OVERWRITE / delete-then-write / swap), never appends blindly. Rerun = same output; the alternative is dedup archaeology after every incident.
- Parameterize by logical date, not wall clock. The run processes
execution_date's data regardless of when it runs;now()inside transforms is how backfills produce garbage. This one rule makes backfilling a loop over dates instead of a special project (see clock-skew for the general principle). - Decide the late-data policy per source. Watermark + lookback (reprocess the trailing N days each run), or land-then-reconcile (raw immutable landing zone, corrections flow to marts). Data always arrives late; a pipeline without a stated policy has the policy "silently wrong".
- Stage raw, transform from staged. Land source data immutable and cheap (see cloud-storage-selection) before any transformation; every downstream table must be rebuildable from the landing zone. When logic changes or a bug is found, you replay from raw instead of begging the source for a re-export.
- Express dependencies as a DAG with real signals. Downstream runs when upstream's partition is complete (sensor on the partition/success marker, not a fixed 3am hope; see pipeline-orchestration). Retries with backoff on transient failures, hard-fail with alerting on logic errors: distinguishing the two is the difference between self-healing and silent corruption.
- Gate outputs with quality checks. Row counts vs history, null/duplicate-key rates, freshness (see data-quality-checks) between transform and publish; fail the run rather than publish a bad partition. Consumers forgive lateness; they do not forgive wrong.
- Backfill as a rehearsed operation. Bounded parallelism (do not flatten the warehouse), ordered by dependency, progress tracked, and cost-estimated before launch: a year's backfill at full cluster is a budget event (see cloud-cost-optimization).
Boundaries
- Streaming pipelines share the principles (idempotency, late data, raw retention) with different machinery; the batch-vs-streaming decision comes first (see batch-vs-streaming).
- Pipelines move and shape data; modeling what the tables mean is warehouse-modeling, and contracts with producers are schema-evolution.
- A pipeline nobody owns rots in a quarter; every DAG carries an owner and an SLA or it is a liability, not an asset.