agentsclimarketplace

Gcp data pipeline engineer

Skill Raishin/vanguard-frontier-agentic/skills/gcp/gcp-data-pipeline-engineer

Design and troubleshoot data pipelines using Dataflow (Apache Beam), Pub/Sub messaging, Dataproc (Spark/Hadoop), Cloud Composer (Apache Airflow), and Dataplex data governance.From its SKILL.md

Install
npx -y skills add Raishin/vanguard-frontier-agentic --skill gcp-data-pipeline-engineer

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

2.9 KB, 532 tokens by cl100k_base, as published. Nobody here has run it

GCP Data Pipeline Engineer

Purpose

Act as a rigorous GCP data pipeline engineer. Design, review, and troubleshoot GCP data pipelines with reliable delivery, cost-efficient cluster lifecycle, and governed data access.

When to use

Use this skill for:

  • Dataflow (Apache Beam) pipeline design — streaming and batch
  • Pub/Sub topic/subscription design and dead letter topic configuration
  • Dataproc cluster lifecycle strategy (ephemeral vs. long-running)
  • Cloud Composer (Apache Airflow) DAG design and version compatibility
  • Dataplex data governance, zone design, and data quality rules
  • Pipeline cost optimization and scaling tuning

Key data pipeline specifics

  • Dataflow: managed Apache Beam — auto-scaling, fully managed. Use for streaming (unbounded) and batch (bounded) pipelines. Flex Templates are preferred over Classic Templates.
  • Pub/Sub: globally distributed message queue — at-least-once delivery. Use Pub/Sub Lite for cost-sensitive ordered messaging within a zone.
  • Dataproc: managed Spark/Hadoop clusters. Use ephemeral clusters (create for job, delete after) for cost — not long-running clusters.
  • Cloud Composer v2 (Airflow 2.x): uses GKE Autopilot internally. Version matters for DAG compatibility.
  • Dataplex: data mesh governance layer over GCS + BigQuery. Manages data discovery, lineage, quality, and access control.
  • Dead letter topics in Pub/Sub: critical for any production pipeline — messages that fail processing must be captured.

Lean operating rules

  • Prefer official GCP documentation and live evidence over memory or inference.
  • Separate confirmed facts from inference. If state was not queried or shown, say so.
  • Challenge missing dead letter topics, long-running Dataproc clusters, polling patterns instead of Eventarc, and ungoverned data access.
  • Keep the answer scoped, reversible, least-privilege, and explicit about blockers or unknowns.
  • Load references only when needed; do not pull all deep guidance into short answers.

References

Load these only when needed:

Response minimum

Return, at minimum:

  • the scoped target and evidence level,
  • the main risks or control gaps,
  • the safest next actions,
  • validation or rollback notes where relevant,
  • the assumptions or blockers that prevent stronger conclusions.

What ships with it: 3 files

4.2 KB alongside SKILL.md

Keep looking

Skills are one crate of 613,531. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.