agentsclimarketplace

Bigquery and dataform platform engineering

Skill vaquarkhan/data-engineering-agent-skills/skills/bigquery-and-dataform-platform-engineering

Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

Install
npx -y skills add vaquarkhan/data-engineering-agent-skills --skill bigquery-and-dataform-platform-engineering

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 21 stars21 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Guides agents through BigQuery- and Dataform-centered data engineering workflows. Use when designing BigQuery physical models, ingestion boundaries, Dataform transformation workflows, slot or cost controls, and platform decisions across BigQuery, Dataflow, Dataproc, and GCP orchestration services.

SKILL.md

3.3 KB, as published. Nobody here has run it

BigQuery And Dataform Platform Engineering

Overview

Use this skill when BigQuery is the center of gravity for analytics engineering and data delivery on GCP. It helps agents decide what should run in BigQuery, what belongs in Dataform, and when the workflow should move to Dataflow, Dataproc, or external orchestration.

When to Use

  • designing or reviewing BigQuery physical models
  • deciding between Dataform, dbt, Dataflow, or Dataproc responsibilities
  • tuning partitioning, clustering, slots, and cost behavior
  • defining ingestion and transformation boundaries on GCP
  • building platform-native analytics workflows around BigQuery

Do not treat BigQuery as a universal default for every preprocessing and orchestration need.

Workflow

  1. Define the workload boundary. Clarify:

    • landing pattern
    • transformation complexity
    • latency requirements
    • governance and regional constraints
    • cost sensitivity

  2. Design BigQuery physical layout intentionally. Cover:

    • partitioning
    • clustering
    • dataset boundaries
    • publish layers
    • data retention and serving expectations

  3. Choose the transformation surface. Consider:

    • Dataform for warehouse-native SQL transformation workflows
    • dbt when the team already standardizes there
    • Dataflow or Dataproc when preprocessing or runtime requirements exceed warehouse-native fit

  4. Define orchestration and operations. Include:

    • where orchestration runs
    • slot and concurrency behavior
    • failure and rerun expectations
    • validation gates before publish

  5. Validate cost and governance behavior. Require:

    • slot or query cost awareness
    • service-account and secret controls
    • policy tags or governance metadata where needed
    • publish-readiness evidence

Common Rationalizations

RationalizationReality
"Everything on GCP should run in BigQuery."Some preprocessing, streaming, or protocol-heavy work belongs in Dataflow, Dataproc, or upstream services.
"Dataform is just a SQL wrapper."It changes how transformation workflows, dependencies, testing, and deployment are managed.
"Partitioning and clustering can be tuned later."Poor physical design often becomes a long-term cost and performance tax.

Red Flags

  • BigQuery physical design is implicit or copied from defaults
  • transformation boundaries between Dataform, Dataflow, and Dataproc are unclear
  • cost or slot behavior is not considered during model design
  • publish and validation behavior is undefined for warehouse-native pipelines
  • governance metadata and access are treated as separate cleanup work

Verification

  • BigQuery physical design choices are explicit and workload-aware
  • Dataform, BigQuery, and other GCP services have clear boundaries
  • Cost, slots, and concurrency implications are documented
  • Governance and access controls are accounted for
  • Publish and validation behavior are explicit before release

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.