agentsclimarketplace

Data lake and zone architecture

Skill vaquarkhan/data-engineering-agent-skills/skills/data-lake-and-zone-architecture

Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

Install
npx -y skills add vaquarkhan/data-engineering-agent-skills --skill data-lake-and-zone-architecture

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 21 stars21 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Guides agents through data lake and zone architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.

SKILL.md

2.9 KB, as published. Nobody here has run it

Data Lake And Zone Architecture

Overview

Use this skill when the storage platform needs structure before pipelines scale into chaos. It helps agents design clear lake zones, dataset boundaries, ownership, lifecycle rules, and publish-safe storage conventions.

When to Use

  • designing a new data lake
  • reorganizing raw, staging, refined, or curated zones
  • defining object storage layout and lifecycle rules
  • separating landing, transformation, and publish responsibilities
  • reducing data swamp behavior in shared lake storage

Do not use this to justify creating extra layers with no operational purpose.

Workflow

  1. Define the lake purpose and consumers. Clarify:

    • source landing needs
    • internal producer teams
    • publish consumers
    • compliance and retention expectations

  2. Define the zone model intentionally. Typical zones include:

    • raw or landing
    • standardized or staging
    • refined or modeled
    • publish or serving

  3. Assign responsibilities to each zone. Decide:

    • who writes to it
    • who reads from it
    • what quality guarantees exist
    • whether mutation is allowed

  4. Design storage conventions. Include:

    • path or catalog naming
    • partition strategy
    • retention lifecycle
    • file-size expectations
    • ownership tags and metadata

  5. Keep publish rules separate from lake convenience. Not every dataset in the lake is ready for shared consumption.

Cross-Cloud Architecture

Use references/cloud-data-engineering-architecture-patterns.md when the task is not only zone design, but choosing the overall cloud architecture pattern across lake, warehouse, lakehouse, streaming, and hybrid shapes.

Common Rationalizations

RationalizationReality
"We can dump everything into one bucket or container and organize later."That is how data lakes turn into data swamps.
"More zones always means better governance."Extra layers without distinct purpose add complexity and slow teams down.
"If the file exists in the lake, it is available for analytics."Raw landing data rarely has the quality or contract guarantees needed for shared use.

Red Flags

  • zone meanings overlap or are undocumented
  • ownership is unclear at the dataset or zone level
  • publish and landing data are mixed together
  • retention, cleanup, or lifecycle policy is absent

Verification

  • The zone model has clear purposes and boundaries
  • Ownership, read/write expectations, and quality guarantees are explicit
  • Storage conventions and lifecycle rules are documented
  • Shared publish datasets are separated from raw landing data

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.