Pipeline orchestration
Skill Amey-Thakur/AI-SKILLS/skills/data-engineering/pipeline-orchestration
Plug-and-play skills and prompts for every AI coding agent
npx -y skills add Amey-Thakur/AI-SKILLS --skill pipeline-orchestrationAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- 19 days oldThe repository was created 19 days ago. New is not bad, but a brand new repository carrying a familiar-sounding name is the shape a typosquat arrives in, and there has been no time for anyone else to find a problem with it.
- 4 stars4 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Orchestrate data DAGs with data-aware scheduling, bounded retries, and SLAs that page the right owner. Use when structuring workflows in an orchestrator or fixing 3am cron archaeology.
SKILL.md
3.0 KB, 672 tokens by cl100k_base, as published. Nobody here has run it
Pipeline orchestration
The orchestrator's job is to run the right tasks after their inputs exist, retry the transient, page on the real, and show a human where things stand. Configuration beyond that is usually complexity debt.
Method
- Schedule on data readiness, not wall clock. Downstream starts when upstream's output partition is complete (dataset/asset triggers, success markers, or sensors with timeouts), not at "03:00 and hope". Clock-scheduled chains break every time an upstream runs long, and the failure lands two teams downstream (see data-pipeline-design).
- Keep tasks coarse and idempotent. One task = one rerunnable unit (build a partition, load a table): reruns are then safe by construction. A DAG of 500 micro-tasks turns the orchestrator into your program and its UI into your debugger; put fine-grained logic in the job's code, not the DAG.
- Separate orchestration from execution. The orchestrator dispatches; heavy compute runs on the warehouse, Spark, or k8s jobs (see kubernetes-workloads). Workers doing the actual transformation on the scheduler's boxes is how one backfill takes down every pipeline's control plane.
- Configure failure behavior per task class. Transient (timeouts, throttling): 2-3 retries, exponential backoff (see timeouts-and-retries). Logic/data errors: no retry, fail loudly with the quality-check context (see data-quality-checks). Everything gets an execution timeout; a hung task holding the DAG open beats no deadline only in the postmortem.
- Express SLAs and route the pages. Per-deliverable freshness SLA (marts ready by 07:00) monitored as lateness alerts to the owning team, not a shared channel of ignorable noise. Alert on: SLA misses, final failures, and sensors timing out; never on retries succeeding.
- Version DAGs as code with tests. DAG definitions in the repo, reviewed, with CI validating imports, dependencies, and a dry-parse (see docs-as-code spirit); parameters over copy-pasted near-identical DAGs. Backfill/rerun procedures are runbook'd per pipeline (see cloud-migration's rehearsal ethic, applied small).
- Stay portable at the edges. Keep business logic in engine-agnostic code/SQL called by thin operators; orchestrator lock-in is real but only bites when logic migrates into plugin-specific operators everywhere.
Boundaries
- Sub-minute or continuous flows are streaming jobs supervised by their own runtime (see batch-vs-streaming); orchestrators are for discrete runs.
- Service-to-service workflows with user-facing latency are application sagas (see saga-pattern), not data orchestration.
- The orchestrator shows task state, not data quality; green DAGs publish bad data happily without the gates from data-quality-checks.