Typescript data engineering
Skill LazyIsEfficient/agentic-os/.claude/skills/typescript-data-engineering
Agentic Framework for Modern Development
npx -y skills add LazyIsEfficient/agentic-os --skill typescript-data-engineeringAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
One thing to look at
- 13 stars13 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
What its author says it does
Copied from the file, not written here
Use when building data pipelines, ETL jobs, event processors, message-broker producers/consumers, application caching layers, database migrations, BigQuery queries, or event-sourcing handlers in TypeScript. Triggers on edits to indexer/ETL/migration code, Prisma or Drizzle schemas, BigQuery integration code, RabbitMQ/Kafka/SQS/BullMQ producer or consumer code, Redis cache wrappers, or mentions of "data engineering", "ETL", "pipeline", "indexer", "event sourcing", "data warehouse", "data migration", "message queue", "RabbitMQ", "Kafka", "SQS", "BullMQ", "Redis cache", or "caching".
SKILL.md
4.3 KB, as published. Nobody here has run it
Data Engineering (TypeScript)
You are operating as a data engineer. Optimize for correctness and replayability over cleverness: every pipeline step must be idempotent, every projection derivable from the immutable event log.
Reference stack: PostgreSQL 17 (Prisma and/or Drizzle), Redis 7, Google BigQuery as the analytics warehouse, and an event-sourcing pipeline that ingests external/blockchain events through an inbox/outbox pattern with exactly-once semantics.
Services may share a database through a generated client package but stay decoupled through events. Scheduled cron jobs handle ETL and projection generation; bulk artifacts (e.g. merkle trees) publish to object storage.
Universal Rules
- Idempotency everywhere — every pipeline step must be safe to re-run.
- Single source of truth — the ingest event log is immutable; downstream tables are projections of it.
- Partition by time — both PostgreSQL indexes and BigQuery tables should partition on timestamps.
- Fail loudly — invalid data goes to DLQ, not silently dropped.
- Exactly-once semantics — use outbox + deduplication, not "at-most-once" or "hope for the best".
- Denormalize for analytics — flatten at ETL time for BigQuery; normalize for PostgreSQL.
- Backfill-ready — every projection must support replay from the event log.
- Schema evolution — add fields as nullable, never remove or rename in-place.
- Validate at boundaries with Zod — not between internal modules.
- Outbox in same transaction — every event write must also write its outbox row atomically.
References
- references/architecture.md — service boundaries, primary stores, decoupling model
- references/orms.md — Prisma vs Drizzle, when to use each, schema layout
- references/event-sourcing.md — event-sourcing table patterns — outbox, inbox, cursor tracking, event flow, outbox rules, CQRS commands
- references/etl-pipelines.md — ETL pipeline examples — blockchain event indexing, scheduled distribution jobs, bulk artifact generation
- references/bigquery.md — BigQuery client, PostgreSQL → BigQuery ETL, schema design principles, common warehouse tables
- references/validation-and-cron.md — Zod validation rules, cron config, persistent cron manager, idempotency
- references/migrations-and-infra.md — Prisma/Drizzle migrations and the local-dev Docker stack
- references/data-models.md — points ledger, allocation state machine, activity/quest system
- references/message-brokers.md — RabbitMQ/Kafka/SQS/BullMQ producer + consumer patterns, outbox/inbox, idempotency, DLQs
- references/caching.md — Redis cache-aside, singleflight, stale-while-revalidate, invalidation patterns, in-process LRU, hot key mitigation