agentsclimarketplace
Sign in

Pipeline orchestrator

Skill baronguyen001/ai-automation-skills/skills/pipeline-orchestrator

Chain a scrape -> AI -> alert pipeline where each stage is a plain callable, with exponential-backoff retry per stage and a JSON state checkpoint between stages so a crashed or rate-limited run resumes instead of restarting. Use for chain pipeline steps, scrape then summarize then notify, retry between stages, resumable pipeline, or orchestrate a job.From its SKILL.md

Install
npx -y skills add baronguyen001/ai-automation-skills --skill pipeline-orchestrator

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.6 KB, 523 tokens by cl100k_base, as published. Nobody here has run it

Pipeline Orchestrator

Use this skill when an automation job is really a chain of steps - scrape, then run it through an AI model, then send an alert - and you want each step to retry on transient failure and the whole run to resume from the last good step. It is a generic, dependency-light runner: stages are ordinary functions, and state is checkpointed to a JSON file so a 429 in the AI step never forces a full re-scrape.

When to invoke

  • User says: "chain these steps" / "scrape then summarize then notify" / "make the pipeline resumable" / "retry between stages"
  • Code in the conversation uses: a script that does fetch -> transform -> deliver in sequence and fails partway through.

When NOT to invoke

  • The work is a single call with no real stages.
  • The user needs a distributed DAG engine (Airflow, Temporal) with workers and a scheduler, not a single-process chain.

Concrete example

User input:

My nightly job scrapes a board, asks Gemini to summarize, then pushes Telegram. If Gemini rate-limits, don't re-scrape.

Output:

# Copy assets/pipeline.py into your project, then:
from pipeline import run_pipeline

def scrape(_):     return fetch_board()          # your scraper
def summarize(rows): return gemini_summary(rows) # your AI step
def alert(text):   return send_telegram(text)    # see telegram-alerter

# checkpoints after each stage; a crash in summarize resumes there, not at scrape
result = run_pipeline([scrape, summarize, alert], state_path=".nightly_state.json")

Pattern to apply

  1. Model the job as an ordered list of single-argument callables; each returns the next stage's input.
  2. Wrap every stage in exponential-backoff retry so a transient error does not kill the run.
  3. Checkpoint {done_index, value} to a JSON file after each successful stage.
  4. On resume, skip completed stages and reuse their cached output.
  5. Keep credentials out of the runner - each stage reads its own env (see [[telegram-alerter]], [[gemini-flash-budget]]).

Reference: assets/pipeline.py.

Source

Distilled from production use across the author's automation projects. v1.0.0. See also: [[cron-dispatch]], [[telegram-alerter]].

→ Build the full runnable bot with Trawlkit.

What ships with it: 1 file

2.5 KB alongside SKILL.md, 1 of them executable

assets/

Gives 0 of the 12 instructions most agent orchestration skills give in 523 tokens

Counted across 848 of the 1,300 authors here whose files we hold, read 2026-09-06

  • Dispatch one agent per independent problem domainin 56 of 848, across 42 files
  • Run full test suite after integrationin 55 of 848, across 42 files
  • Verify fixes do not conflictin 40 of 848, across 32 files
  • Review each summary when agents returnin 40 of 848, across 31 files
  • Write a handoff document summarising the current conversationin 30 of 848, across 25 files
  • Reference existing artifacts by path or URLin 26 of 848, across 24 files
  • Give each agent a specific scopein 19 of 848, across 10 files
  • Give each agent a clear goalin 19 of 848, across 10 files
  • Include a suggested skills section in the documentin 18 of 848, across 16 files
  • Tailor the doc to the user argumentsin 18 of 848, across 15 files
  • Issue all subagent dispatches in the same responsein 17 of 848, across 11 files
  • Use git worktrees for isolationin 17 of 848, across 8 files

Said here and by no other author read

  • Model the job as an ordered list of single-argument callables
  • Wrap every stage in exponential-backoff retry
  • Skip completed stages on resume and reuse cached output

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.