agentsclimarketplace

Verify pipeline

Skill pol-cc/agentic-data-engineer/skills/verify-pipeline

A Claude Code harness that turns a session into an agentic data engineer for SMBs — packaged as an installable plugin, built from a skillpack of skills that stand up a cheap, self-hostable Modern Data Stack (Tailscale + dlt + BigQuery + dbt + optional MCP), end-to-end and headless.

Install
npx -y skills add pol-cc/agentic-data-engineer --skill verify-pipeline

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Run a full health check across the MDS pipeline: ingestion (dlt/Airbyte) load status, BigQuery freshness per source, ingest reconciliation (source-vs-destination row counts), dbt model freshness, MCP server health, and raw-vs-staging row count integrity. Invoke when the user wants to confirm the pipeline is healthy or asks 'is everything working?'

SKILL.md

3.6 KB, as published. Nobody here has run it

verify-pipeline

Status: v0.10.0 — references written; read-only health check operational. Ingest reconciliation is now a first-class layer (source-vs-destination row counts, dlt _dlt_loads freshness, sequence/gap checks) — mandatory after every dlt load to catch the silent data gap a mis-set incremental cursor leaves without crashing.

What this skill does

Runs deterministic checks across every layer of the MDS and produces a one-page report. Read-only — never modifies state. Safe to invoke at any time.

Preflight

if [ ! -f .agentic-data-engineer.json ]; then
  echo "[abort] not a managed MDS deployment"
  exit 1
fi

Checks performed

LayerCheckPass criterion
Tailscaletailscale status on the VPS via SSHVPS reachable, all nodes online
Ingestion (dlt/Airbyte)dlt _dlt_loads last-load status + age per source (or Airbyte GET /jobs when stack.ingest == "airbyte")Latest load completed within freshness_thresholds.green_hours (default 26h)
BigQuery raw__TABLES__ modification time per raw datasetUpdated within green_hours
Ingest reconciliationSource-vs-destination row count per source; dlt _dlt_loads status; sequence/gap check on monotonic keysDestination matches source within reconciliation_tolerance (default 0); no sequence gaps; latest _dlt_loads.status = 0
BigQuery integrityRow count raw.<table> vs staging.stg_<table>Difference within 0.5% (or configured threshold)
dbttarget/run_results.json from last cron run via SSHAll models success, run completed within green_hours
MCP (if configured)GET /health on the MCP server endpointReturns 200

Reconciliation is the ingest-layer check dbt tests don't cover. dbt tests validate the transform (raw → staging → marts); reconciliation validates the ingest (source → raw). It is mandatory after every dlt load because dlt's failure mode is silent — a mis-set incremental cursor or broken paginator leaves a data gap without crashing, so freshness looks green while rows are missing. Only counting source against destination catches it. See references/health-checks.md section 4.

Output

A markdown report with: per-source traffic-light status, last successful sync timestamp, dbt model freshness, integrity warnings, and a one-line global verdict.

The skill never auto-fixes. If a check fails, it points the user at troubleshoot.

References

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.