Data pipeline

Skill christopherlouet/claude-base/.claude/skills/data-pipeline

ETL/ELT pipeline design. Trigger when the user wants to create data flows, transformations, or orchestration.From its SKILL.md

Install

npx -y skills add christopherlouet/claude-base --skill data-pipeline

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

5 stars5 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.7 KB, 375 tokens by cl100k_base, as published. Nobody here has run it

Data Pipeline

ETL vs ELT

Pattern	When to use
ETL	Complex transformation, sensitive data
ELT	Big data, cloud DW (BigQuery, Snowflake)

Airflow DAG

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-team',
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'daily_etl',
    default_args=default_args,
    schedule_interval='0 2 * * *',
    start_date=datetime(2024, 1, 1),
    catchup=False,
) as dag:

    extract = PythonOperator(
        task_id='extract',
        python_callable=extract_from_source,
    )

    transform = PythonOperator(
        task_id='transform',
        python_callable=transform_data,
    )

    load = PythonOperator(
        task_id='load',
        python_callable=load_to_warehouse,
    )

    extract >> transform >> load

dbt Transformation

-- models/staging/stg_orders.sql
{{ config(materialized='view') }}

SELECT
    id AS order_id,
    customer_id,
    order_date,
    CAST(total AS DECIMAL(10,2)) AS total_amount
FROM {{ source('raw', 'orders') }}
WHERE order_date >= '2023-01-01'

Data Quality

def validate_data(df):
    assert df['order_id'].is_unique, "Duplicate IDs"
    assert df['amount'].ge(0).all(), "Negative amounts"
    assert df['customer_id'].notna().all(), "Null customers"

What ships with it: 1 file

3.8 KB alongside SKILL.md

examples/

etl-pipeline.md3.8 KB