Data pipelines
Skill Methasit-Pun/data_engineer_claude_skills/05-etl-build/data-pipelines
Practical guides, prompts, and Python code for applying Anthropic's Claude Skills to data engineering and pipeline automation
npx -y skills add Methasit-Pun/data_engineer_claude_skills --skill data-pipelinesAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Umbrella skill for moving data from source to destination — end-to-end ETL/ELT design, DAG orchestration, real-time streaming, and system-to-system migration. Use this whenever the user is building, scheduling, debugging, or migrating a pipeline and it isn't yet clear which sub-area dominates. This skill ROUTES to the focused sub-skills (pipeline-design, orchestration-patterns, streaming-patterns, data-migration) and pulls in more than one when a task spans them (e.g. "design a streaming pipeline and orchestrate its backfill"). Trigger on: new ingestion job, batch vs. streaming choice, Airflow/Prefect/Dagster DAGs, CDC, backfill, cutover, dual-write, or "move data from X to Y".
SKILL.md
2.6 KB, as published. Nobody here has run it
Data Pipelines (Router)
This is a router skill. It groups the four skills that deal with getting data from a source into a destination and keeping it flowing. Diagnose which sub-area(s) the task touches, then invoke the matching sub-skill(s) with the Skill tool. For a task that spans areas, invoke several and combine their guidance.
How to route
| If the task is about… | Invoke sub-skill |
|---|---|
| Designing a new ingestion job end-to-end: extraction strategy, idempotency, load patterns, raw/staging/marts layers, ELT vs ETL | pipeline-design |
| Scheduling multi-step pipelines, task dependencies, retries, SLAs, sensors, failure recovery, choosing between Airflow/Prefect/Dagster | orchestration-patterns |
| Real-time / near-real-time processing, Kafka/Flink/Kinesis/Spark Structured Streaming, consumer lag, windowing, exactly-once, late data | streaming-patterns |
| Moving between systems safely: cutover planning, backfill, dual-write, shadow reads, validation, rollback, retiring a legacy pipeline | data-migration |
Routing rules
- Default new-pipeline questions to
pipeline-designfirst — it frames source shape, volume, change pattern, and downstream contract, which the others build on. - Batch vs. streaming undecided? Invoke
streaming-patterns(it contains the stream-vs-batch decision) before committing to an architecture. - Anything scheduled with >2 dependent steps also pulls in
orchestration-patterns. - Replacing or moving off an existing system always pulls in
data-migration, usually alongsidepipeline-design. - Invoke via the Skill tool by name, e.g. call
Skill(skill="orchestration-patterns"). Combine outputs; don't paraphrase from memory.
Related groups
- Modeling the data once it lands → [[data-modeling]]
- Validating and governing it → [[data-reliability]]
- Where it runs and what it costs → [[cloud-data-infra]]