Engineering data pipelines
Data engineering knowledge reference covering Airflow, Dagster, Kafka Streams, Flink, dbt, and data quality patterns. Use when building data pipelines, ETL workflows, stream processing, or data quality checks.From its SKILL.md
npx -y skills add telagod/code-abyss --skill engineering-data-pipelinesAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
1.3 KB, 390 tokens by cl100k_base, as published. Nobody here has run it
数据工程域 · Data Engineering
判断先于执行：决定「是否做 / 选什么 / 如何取舍」（栈、方案、架构、权衡）前，先读领域判断内核
skills/_kernel/backend/SKILL.md——它管 judgment，本秘典管 execution；冲突时以内核判断为准。
编排：Airflow(调度) | Dagster(资产) | Prefect(现代流)
流处理：Kafka Streams(嵌入式) | Flink(集群) | Spark Streaming
质量：Great Expectations | dbt tests | Soda Core
编排检查项
幂等(UPSERT/分区覆盖) | 增量(WHERE updated_at > last_run) | 事件驱动触发 | 跨 DAG 依赖 | 数据血缘(ref()/Asset deps)
流处理检查项
时间语义选择 | Watermark 乱序容忍 | 状态 TTL 防膨胀 | Checkpoint 间隔 | 端到端 Exactly-Once | 背压监控
质量检查项
分层验证(源→转换→目标) | 完整性+准确性+一致性 | 及时性阈值 | 加权评分 | 告警(Slack/PagerDuty)
工具对比、API 用法、质量维度详见 references/details.md
What ships with it: 1 file
1.8 KB alongside SKILL.md
references/
- details.md1.8 KB
Gives 0 of the 12 instructions most data pipelines skills give in 390 tokens
Counted across 149 of the 156 authors here whose files we hold, read 2026-09-06
- Run a safe catch-up or sample benchmarkin 13 of 149, across 4 files
- Rerun final accounting after the codified path executesin 13 of 149, across 4 files
- Move compute to where the data already isin 13 of 149, across 4 files
- Batch small files, requests, and writesin 13 of 149, across 4 files
- Use manifests or checkpoints to skip completed filesin 13 of 149, across 4 files
- Measure backlog across files, rows, and timestampsin 13 of 149, across 4 files
- Codify the path as a CLI or scheduled jobin 12 of 149, across 3 files
- Promote only the fastest correctness-preserving pathin 10 of 149, across 3 files
- Separate the bottleneck categories before optimizingin 10 of 149, across 3 files
- Prefer warehouse-native scans, joins, and appendsin 9 of 149, across 2 files
- Make writes idempotent through keys, manifests, or replaceable stagingin 9 of 149, across 2 files
- Retry failures with exponential backoffin 9 of 149, across 7 files
Said here and by no other author read
- Read the judgment kernel before deciding
- Process incrementally since the last run
- Track data lineage
- Choose time semantics deliberately
- Set watermarks to tolerate out-of-order events
- Set state TTL to prevent state bloat
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.