agentsclimarketplace

Pipeline orchestrator

Skill baronguyen001/ai-automation-skills/skills/pipeline-orchestrator

8 production-tested Claude skills: automation, Gemini cost/structured output, OSS bounty scouting, ML validation.

Install
npx -y skills add baronguyen001/ai-automation-skills --skill pipeline-orchestrator

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Chain a scrape -> AI -> alert pipeline where each stage is a plain callable, with exponential-backoff retry per stage and a JSON state checkpoint between stages so a crashed or rate-limited run resumes instead of restarting. Use for chain pipeline steps, scrape then summarize then notify, retry between stages, resumable pipeline, or orchestrate a job.

SKILL.md

2.6 KB, 523 tokens by cl100k_base, as published. Nobody here has run it

Pipeline Orchestrator

Use this skill when an automation job is really a chain of steps - scrape, then run it through an AI model, then send an alert - and you want each step to retry on transient failure and the whole run to resume from the last good step. It is a generic, dependency-light runner: stages are ordinary functions, and state is checkpointed to a JSON file so a 429 in the AI step never forces a full re-scrape.

When to invoke

  • User says: "chain these steps" / "scrape then summarize then notify" / "make the pipeline resumable" / "retry between stages"
  • Code in the conversation uses: a script that does fetch -> transform -> deliver in sequence and fails partway through.

When NOT to invoke

  • The work is a single call with no real stages.
  • The user needs a distributed DAG engine (Airflow, Temporal) with workers and a scheduler, not a single-process chain.

Concrete example

User input:

My nightly job scrapes a board, asks Gemini to summarize, then pushes Telegram. If Gemini rate-limits, don't re-scrape.

Output:

# Copy assets/pipeline.py into your project, then:
from pipeline import run_pipeline

def scrape(_):     return fetch_board()          # your scraper
def summarize(rows): return gemini_summary(rows) # your AI step
def alert(text):   return send_telegram(text)    # see telegram-alerter

# checkpoints after each stage; a crash in summarize resumes there, not at scrape
result = run_pipeline([scrape, summarize, alert], state_path=".nightly_state.json")

Pattern to apply

  1. Model the job as an ordered list of single-argument callables; each returns the next stage's input.
  2. Wrap every stage in exponential-backoff retry so a transient error does not kill the run.
  3. Checkpoint {done_index, value} to a JSON file after each successful stage.
  4. On resume, skip completed stages and reuse their cached output.
  5. Keep credentials out of the runner - each stage reads its own env (see [[telegram-alerter]], [[gemini-flash-budget]]).

Reference: assets/pipeline.py.

Source

Distilled from production use across the author's automation projects. v1.0.0. See also: [[cron-dispatch]], [[telegram-alerter]].

→ Build the full runnable bot with Trawlkit.

What ships with it: 1 file

2.5 KB alongside SKILL.md, 1 of them executable

assets/

Gives 0 of the 12 instructions most agent orchestration skills give in 523 tokens

Counted across 742 of the 995 authors here whose files we hold, read 2026-08-07

  • reference existing artifacts by path or URLin 53 of 742, across 25 files
  • run the full test suite after integrating changesin 51 of 742, across 19 files
  • dispatch one agent per independent problem domainin 50 of 742, across 17 files
  • verify fixes do not conflictin 45 of 742, across 13 files
  • include a suggested skills section in the documentin 45 of 742, across 17 files
  • redact sensitive informationin 41 of 742, across 11 files
  • save to the temporary directory of the operating systemin 39 of 742, across 10 files
  • tailor the document to user-provided focus argumentsin 39 of 742, across 9 files
  • spot check agent changes for systematic errorsin 34 of 742, across 7 files
  • write a handoff document summarising the current conversationin 31 of 742, across 6 files
  • Assign each agent a specific scopein 23 of 742, across 8 files
  • provide specific scope and clear goalin 23 of 742, across 5 files

Said here and by no other author read

  • model the job as ordered single-argument callables
  • wrap every stage in exponential-backoff retry
  • checkpoint state to a JSON file after each stage
  • skip completed stages on resume
  • reuse cached output of completed stages on resume
  • keep credentials out of the runner

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.