Pipelines & ETL
567 rows from 163 repositories
Moving data from where it is to where it is useful.
What Pipelines & ETL skills agree on
200 skill files read, by 149 of the 156 authors on this shelf whose files we hold, 2026-09-06
The middle one of the 42 measured here is ~1.7k tokens long, counted with cl100k_base
Counted by distinct author, so one author publishing three of these counts once. Where a claim sits in fewer files than authors, that is said: a claim held by forty authors across three files is one file people copied, not forty people who agreed. Near-identical wordings are grouped and the other wordings are shown, so the grouping is yours to check.
What they tell the agent to do
- Run a safe catch-up or sample benchmark13 of 149 in 4 files
- Rerun final accounting after the codified path executes13 of 149 in 4 files
- Move compute to where the data already is13 of 149 in 4 files
- Batch small files, requests, and writes13 of 149 in 4 files
- Use manifests or checkpoints to skip completed files13 of 149 in 4 filesalso worded as Skip completed files using manifests or checkpoints
- Measure backlog across files, rows, and timestamps13 of 149 in 4 filesalso worded as Measure backlog files, rows, and timestamps before optimizing; Measure backlog files, rows, timestamps, and unprocessed counts
- Codify the path as a CLI or scheduled job12 of 149 in 3 filesalso worded as Codify the path as a CLI, job, or runbook
- Promote only the fastest correctness-preserving path10 of 149 in 3 files
- Separate the bottleneck categories before optimizing10 of 149 in 3 filesalso worded as Separate bottleneck categories before optimizing; Separate the bottleneck types before optimizing.
- Retry failures with exponential backoff9 of 149 in 7 filesalso worded as Retry transient failures with backoff; Retry with exponential backoff and jitter
- Prefer warehouse-native scans, joins, and appends9 of 149 in 2 files
- Make writes idempotent through keys, manifests, or replaceable staging9 of 149 in 2 filesalso worded as Make writes idempotent through unique keys or staging
- Route invalid records to a dead letter queue8 of 149 in 5 filesalso worded as Route unprocessable records to a dead letter queue; Route invalid records to a dead letter table
- Keep raw, derived, and serving tables separately accountable.7 of 149 in 2 files
- Make every pipeline idempotent7 of 149also worded as Make pipelines idempotent; Make every pipeline idempotent, observable, and failure-handled
What they tell it not to do
- Do not skip failed files silently13 of 149 in 4 files
- Do not mix backfill status with live-tail freshness13 of 149 in 4 files
- Do not delete raw data to improve metrics13 of 149 in 4 filesalso worded as Do not delete raw data to flatter a metric.
- Do not call completion until tables and manifest agree13 of 149 in 4 filesalso worded as Do not call it complete until tables and manifest agree; Do not call the pipeline complete until tables and manifest agree
- Never hardcode credentials8 of 149also worded as Don't hardcode credentials in profiles.yml; Never miss a hardcoded credential
- Do not hardcode model names5 of 149also worded as don't hardcode model names; Never hardcode table names in model SQL
- Do not use for ML development without pipelines4 of 149 in 2 filesalso worded as Do not use ml for ETL pipeline design
- Never skip data quality checks4 of 149also worded as Never skip data quality for small pipelines; Do not skip data quality checks at ingestion
- Do not use for exploratory data analysis only3 of 149 in 1 file
- Do not use without access to data sources3 of 149 in 1 file
What they expect to be installed
- dbt42 of 149
- Airflow21 of 149 in 19 files
- Kafka17 of 149 in 14 files
- Great Expectations15 of 149 in 10 files
- Spark13 of 149 in 10 files
- python11 of 149
- docker10 of 149 in 9 files
- Prefect10 of 149 in 8 files
- Apache Airflow9 of 149 in 7 files
- curl8 of 149
What they ask it to produce
- Codified throughput path as CLI, job, or runbook13 of 149 in 4 filesalso worded as codified path as CLI, job, or runbook; Codified fast path as CLI, job, or runbook
- dbt model SQL files10 of 149also worded as dbt models; dbt SQL models
- Hard accounting block with counts, runtime, and correctness gate7 of 149 in 2 filesalso worded as hard accounting block of counts, tail, runtime, correctness gate
- Data contracts7 of 149 in 4 filesalso worded as data contract; Data contracts table
- Operational runbooks6 of 149 in 4 files
- Airflow DAGs6 of 149 in 5 filesalso worded as Airflow DAG file; Airflow DAG
- schema.yml with tests and descriptions5 of 149also worded as _models.yml entries with tests and descriptions; schema.yml with tests and documentation
- Data pipelines4 of 149 in 2 filesalso worded as data pipeline design
When Pipelines & ETL authors say to reach for one
The situations these authors wrote into their own files, counted out of the same 149 authors, with the skills that name each one
- Large ingestion, backfill, export, ETL, or warehouse loading must be faster13 of 149 in 4 files
- Creating data pipelines10 of 149 in 9 files
- Bottleneck is moving, transforming, or saving lots of data7 of 149 in 3 files
- User asks for a data pipeline or ETL7 of 149
- Designing batch or streaming data pipelines6 of 149 in 4 files
- Data engineer
- Data pipeline design
- and 2 more on this shelf
- Setting up Airflow in production5 of 149
- Airflow dag patterns
- and 3 more on this shelf
- Orchestrating workflows4 of 149
- Airflow dag patterns
- and 3 more on this shelf
How Pipelines & ETL skills are built
481 skill directories by 150 authors, read from their repositories’ own file trees 2026-08-05
The middle bundle among those shipping files is 2 files, 13.6 KB beside SKILL.md
Counted by distinct author, same as above, so one author publishing forty template copies counts once. SKILL.md itself is not counted as a file, so a single-file skill is one where that file is the whole skill.
The shape
- SKILL.md is the whole skill58 of 150 authors, 261 of 481 skills
- files ship beside it92 of 150 authors, 220 of 481 skills
- executable scripts ship inside30 of 150 authors, 45 of 481 skills
The folders they converge on
- references/58 of 150 authors, 114 of 481 skills
- scripts/23 of 150 authors, 32 of 481 skills
- resources/9 of 150 authors, 30 of 481 skills
- assets/7 of 150 authors, 10 of 481 skills
- examples/7 of 150 authors, 10 of 481 skills
- agents/6 of 150 authors, 17 of 481 skills
tt-a1i/archify/archify Skill
10,485★ repoAgent skill for beautiful, verifiable architecture, workflow, sequence, data-flow, and lifecycle diagrams—self-contained HTML with motion and crisp export.
lensesio/agentic-engineering-for-apache-kafka/skills/kafka-connector-review Skill
56★ repoAI agent skills for building, operating and troubleshooting Apache Kafka applications. Topic audit, consumer lag, schema review, security, connectors and DLQ
abhishekbhakat/airflow-mcp-server/io.github.abhishekbhakat/airflow-mcp-server MCP server
33★ repoAn MCP server for Apache Airflow
Airflow and workflow orchestration
vaquarkhan/data-engineering-agent-skills/skills/airflow-and-workflow-orchestration Skill
29★ repoProduction-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
Chipagosfinest/enterprise-team/enterprise-team/skills/data-catchall Skill
no license18★ repoHire a whole company with one plugin. 75 specialized AI agents across Engineering, Product, Infrastructure, Data, Security, Marketing, Sales, Finance, Legal, and People.
G1Joshi/Agent-Skills/skills/ai-ml/airflow Skill
12★ repoA comprehensive skill catalog for AI agents
zenml-io/skills/skills/zenml-airflow-migration/skills/airflow-migration Skill
6★ repoAI coding agent skills for ZenML MLOps workflows — quick wins, pipeline setup, and more
Incident Response MCP for Apache Airflow
madamak/apache-airflow-mcp-server/io.github.madamak/apache-airflow-mcp-server MCP server
2★ repoDiagnose Airflow failures from UI links with bounded logs and optional recovery actions.
sanjay-amu/kafka-sentinel-mcp/io.github.sanjay-amu/kafka-sentinel-mcp MCP server
2★ repoRead-only Kafka observability MCP server: cluster health, consumer lag, and replay-readiness.
us-all/airflow-mcp-server/io.github.us-all/airflow MCP server
2★ repoAirflow MCP — list DAGs/runs/task instances, tail logs, trigger and clear (write-gated)
pol-cc/agentic-data-engineer/skills/add-dbt-model Skill
1★ repoA Claude Code harness that turns a session into an agentic data engineer for SMBs — packaged as an installable plugin, built from a skillpack of skills that stand up a cheap, self-hostable Modern Data Stack (Tailscale + dlt + BigQuery + dbt + optional MCP), end-to-end and headless.
pedroberaldo87/pedro-plugins/plugins/archify/skills/archify Skill
no license0★ repoMarketplace privado de 17 plugins (skills, hooks e automações) pessoais para Claude Code.
alyshahendo/dbt-debugger/.claude/skills/dbt-debugger Skill
0★ repoAn open source Claude Code skill and CLI that traces dbt pipeline failures back to root cause, distinguishing genuine breaks from downstream models skipped as collateral. No warehouse connection needed.
danielvogler/ai-skills-library/.agents/skills/adding-dbt-unit-test Skill
0★ repoCentralized manifest for AI agent skills. Synced via ai-agent-skills.
Agent skill enterprise data ecosystems
k-sandhu/agent-skill-enterprise-data-ecosystems Skill
0★Claude agent skill that generates realistic fictional enterprise data ecosystems — multi-system schemas, populated SQLite, SQL lineage, controlled DQ imperfections, and a generated read-only MCP server.
glassflow/agent-skills/skills/create-pipeline Skill
0★ repoSkills for driving GlassFlow ETL pipelines from AI coding assistants
satishTheLegend/ingestion-pipeline-architect Skill
0★Design a resilient data ingestion/ETL pipeline from source to warehouse: contracts, idempotency, quality gates, lineage, backfill, and run-time observability.
datanika-io/datanika-core/io.datanika/datanika-mcp MCP server
0★ repoRead-only-by-default MCP for Datanika: browse data, run dbt transforms, manage ELT pipelines.
Aguantar/kafka-mcp-server/io.github.Aguantar/kafka-dataops-mcp MCP server
0★ repoDataOps Kafka MCP server with consumer lag diagnosis and broker monitoring
dbsectrainer/mcp-data-pipeline-connector/io.github.dbsectrainer/mcp-data-pipeline-connector MCP server
0★ repoUniversal data connector for CSV, Postgres, and REST APIs via DuckDB
us-all/dbt-mcp-server/io.github.us-all/dbt MCP server
0★ repodbt MCP — manifest/run_results/sources/catalog parsing + DQ result tables (BigQuery/Postgres)
Kaelio/ktx/packages/cli/src/skills/dbt_ingest Skill
1,540★ repoktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company
ancoleman/ai-design-components/skills/ai-data-engineering Skill
503★ repoComprehensive UI/UX and Backend component design skills for AI-assisted development with Claude
AltimateAI/data-engineering-skills/skills/dbt/creating-dbt-models Skill
118★ repoSkills related to Data Engineering Work for Claude Code
lensesio/agentic-engineering-for-apache-kafka/skills/kafka-consumer-lag Skill
56★ repoAI agent skills for building, operating and troubleshooting Apache Kafka applications. Topic audit, consumer lag, schema review, security, connectors and DLQ
Benkapner/claude-code-basecamp/skills/data-pipeline-patterns Skill
14★ repoA meta-repository for Claude Code users that includes workspace setup (skills, commands, hooks)
risingwavelabs/agent-skills/skills/risingwave Skill
no license9★ repoAgent Skills for RisingWave.
cskwork/supergoal-skill/templates/archify Skill
4★ repoOne objective in, a verified result out. An agent skill that runs a full, gated dev process with expert subagents and refuses to declare done until a machine-checkable gate passes. Bilingual (EN/한국어) onboarding & live walkthrough
satbirbhbc-ux/ai-coding-principles/ddia-principles Skill
3★ repoBuild AI coding principles for Claude Code, with discipline rules and system design guides to improve code quality and reduce bad patterns
samarth777/modal-skills/skills/data-pipeline Skill
1★ repoThis skill teaches Claude how to effectively build and deploy applications on Modal's serverless platform.
NeuralMedic-DE/claude-skills/dbt-data-quality-gate Skill
0★ repoVerified-gate Claude Code skills that run their checks and prove the result — by NeuralMedic. Web, accessibility, healthcare & compliance, data, IaC, Odoo.
chrisVillanueva/ai-skills-registry/data-engineering/nonfunctional-requirements Skill
no license0★ repoA registry of agent-loadable skills that distill technical knowledge and expertise from prior large-scale, enterprise engagements. Focus given to decision-making frameworks, trade-offs analysis, and addresses conditions that require subtle technical judgments. The Agent Skills open standard is used to develop all skills.
kozen-labs/agentic/.agents/skills/kozen-etl-mk Skill
0★ repoAgentic Repository: Skills, Sub-Agents, Hooks, Context, etc.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/airflow-dag-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/api-integration Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/data-engineer Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
Data engineering data pipeline
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/data-engineering-data-pipeline Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills-claude/skills/snowflake-development Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/airflow-dag-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/api-integration Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/data-engineer Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
Data engineering data pipeline
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/data-engineering-data-pipeline Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-awesome-skills/skills/snowflake-development Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-aas-data-analytics/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-aas-data-engineering-platform/skills/airflow-dag-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-aas-data-engineering-platform/skills/data-engineer Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-aas-data-engineering-platform/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-data-engineering/skills/airflow-dag-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-data-engineering/skills/data-engineer Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-data-engineering/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/plugins/agentic-bundle-integration-apis/skills/api-integration Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/skills/airflow-dag-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/skills/api-integration Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/skills/data-engineer Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
Data engineering data pipeline
sickn33/agentic-awesome-skills/skills/data-engineering-data-pipeline Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/skills/dbt-transformation-patterns Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
sickn33/agentic-awesome-skills/skills/snowflake-development Skill
44,658★ repoAAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 1,987+ agentic skills. Includes CLI, local MCP, catalog, plugins, and Workbench.
wshobson/agents/plugins/data-engineering/skills/airflow-dag-patterns Skill
38,701★ repoMulti-harness agentic plugin marketplace for Claude Code, Codex CLI, Cursor, OpenCode, GitHub Copilot, and Gemini CLI