Ship & operate
32,413 rows, by stacks then stars
1,245 of these skill files have been read, by 1,077 distinct authors, and what they tell an agent counted. What Ship & operate authors agree on, and what they forbid
Infra kubernetes storage review
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_kubernetes_storage_review Skill
no license15★ repoKubernetes storage review — PVC lifecycle (provisioning/binding/reclaim), StorageClass selection (SSD/HDD/NVMe), dynamic vs static provisioning, StatefulSet volume management, volume performance tuning (ReadWriteMany vs ReadWriteOnce), CSI drivers (AWS EBS/GCP PD/Ceph/Longhorn), PVC resize, snapshot and backup, storage class access modes, orphaned PVs detection, shared storage patterns (NFS/CephFS)
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_multi_cloud_governance Skill
no license15★ repoMulti-cloud governance — cloud-agnostic data platform patterns, federated identity (OIDC/SAML between AWS/GCP/Azure), Terraform multi-cloud modules, cross-cloud data replication (S3↔GCS/Azure), unified cost management (FinOps Foundation framework), cloud-agnostic observability (OpenTelemetry), policy enforcement (OPA Gatekeeper across clouds), disaster recovery cross-cloud, vendor lock-in avoidance (open formats Iceberg/Parquet), centralized secrets management (HashiCorp Vault)
Infra observability stack review
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_observability_stack_review Skill
no license15★ repoObservability stack review — three pillars (metrics/logs/traces), Prometheus+Grafana+Loki+Tempo (LGTM stack), OpenTelemetry Collector as universal agent, cardinality management, log aggregation patterns (Fluent Bit→Loki), distributed tracing (Jaeger/Tempo), alerting pipeline (Alertmanager routing), SLO definition and error budget tracking, data platform observability (pipeline freshness/throughput/error rate), observability as code (Grafana provisioning)
Infra opentelemetry instrumentation
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_opentelemetry_instrumentation Skill
no license15★ repoOpenTelemetry instrumentation for data platforms — Python auto-instrumentation (opentelemetry-bootstrap), manual span creation for ETL tasks, trace context propagation across Airflow→Spark→dbt, OTLP exporter configuration, W3C TraceContext headers in Kafka messages, resource attributes (service.name/environment/version), Baggage for partition metadata propagation, Collector pipeline (tail sampling), correlation between traces/logs/metrics, Grafana Tempo integration
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_prometheus_optimization Skill
no license15★ repoPrometheus optimization — recording rules (level:metric:operations naming), cardinality explosion diagnosis (tsdb status API), scrape interval tuning, remote write configuration (batch size/queue capacity), Thanos/Mimir for long-term retention, federation vs remote write, AlertManager routing trees (inhibition/silencing), absent() alerts for missing series, relabeling to drop expensive labels, TSDB compaction, ServiceMonitor vs PodMonitor patterns
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_rbac_audit Skill
no license15★ repoRBAC audit for data platforms — Kubernetes RBAC (ClusterRole/Role/Binding audit), AWS IAM permission boundary and policy review (AWS IAM Access Analyzer), GCP IAM audit (principle of least privilege/owner role detection), database RBAC (PostgreSQL/Trino/ClickHouse role hierarchies), Airflow RBAC (role-based menu/DAG access), dbt permissions (warehouse roles), service account proliferation detection, wildcard permission detection, privilege escalation paths
Infra secrets management review
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_secrets_management_review Skill
no license15★ repoSecrets management review — HashiCorp Vault (KV v2/dynamic credentials/lease renewal/audit log), External Secrets Operator (Vault/AWS SSM/GCP Secret Manager → K8s secrets), secret rotation strategies (database dynamic credentials/TLS cert-manager), detecting secrets in code (gitleaks/truffleHog/Semgrep), Airflow connections from Vault backend, dbt profiles without hardcoded credentials, CI/CD secrets (GitHub OIDC/GitLab CI variables), no plaintext secrets in logs/configs
Infra streaming reliability review
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_streaming_reliability_review Skill
no license15★ repoStreaming pipeline reliability — exactly-once semantics (idempotent producer/transactional API/read_committed/Kafka Streams EOS), at-least-once patterns with idempotent consumers, dead letter queue (DLQ) design with Kafka Connect SMT/custom error handler, poison message handling, reprocessing strategy, watermarks and late data handling, checkpoint/savepoint for Flink/Spark Streaming, consumer group rebalance storm prevention, end-to-end latency SLO monitoring, streaming reliability checklist
Infra terraform cost estimator
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_terraform_cost_estimator Skill
no license15★ repoTerraform cost estimation — Infracost breakdown and diff commands (monthly cost per resource), PR cost comments (before/after change delta), budget thresholds (block PRs over cost limit), usage files for variable quantities (S3 GB/requests), AWS pricing data sources, cost tagging strategy, OpenTofu/Terraform cost attribution, FinOps tagging policy enforcement, multi-environment cost comparison (dev vs prod)
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_terraform_review Skill
no license15★ repoTerraform code review — module structure (main/variables/outputs/modules layout), variable validation blocks, type constraints, sensitive variables, locals vs variables, resource naming conventions, provider version pinning, data sources vs resources, module composition patterns, state management (remote backend S3/GCS), workspace strategy, dependency management (depends_on vs implicit), terraform fmt/validate/tflint checks, DRY with Terragrunt
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/infra_terraform_security_scan Skill
no license15★ repoTerraform security scanning — tfsec static analysis (AWS/GCP/Azure misconfigurations), Checkov IaC policy checks (750+ rules, CIS benchmarks), tflint security rules, KICS scanner, S3 encryption/public access/versioning checks, IAM least-privilege patterns, security group rule review (0.0.0.0/0), KMS key rotation, VPC flow logs, CloudTrail enabled, pre-commit hooks, SARIF output for GitHub Security tab, policy-as-code with OPA/Sentinel
Platform engineering agentic control plane
ivanshamaev/de-agent-skills/group_skills/infra_dataops_group_skills/platform_engineering_agentic_control_plane Skill
no license15★ repoAgentic control plane for data platform — MCP (Model Context Protocol) server exposing platform tools to AI agents (FastMCP/Python SDK), natural language platform operations (trigger DAG/create topic/run dbt/query Trino via LLM), Claude-based platform assistant with tool use, multi-agent platform governance (audit agent/cost agent/reliability agent), MCP server deployment (Docker/Kubernetes), tool authorization and audit logging, agentic workflow patterns (plan-execute-verify), platform chat interface
Airflow starrocks cdc orchestrator
ivanshamaev/de-agent-skills/group_skills/starrocks_group_skills/airflow_starrocks_cdc_orchestrator Skill
no license15★ repoAirflow CDC orchestration for StarRocks — watermark-based incremental sync, Routine Load health check DAG, Flink job submission via REST API, offset lag alerting, dead letter queue reprocessing, multi-table incremental sync with dependency graph, schema change detection and pipeline pause/resume
Dbt starrocks production readiness
ivanshamaev/de-agent-skills/group_skills/starrocks_group_skills/dbt_starrocks_production_readiness Skill
no license15★ repodbt + StarRocks production readiness — slim CI with state:modified+, manifest.json artifact storage and --defer for dev, dbt Cloud vs dbt Core deployment, RBAC for dbt users in StarRocks, secrets management for profiles.yml, dbt run/test exit codes in CI, breaking change detection, scheduled dbt runs via Airflow/dbt Cloud, dbt docs generation and hosting, model versioning
Starrocks admin cluster health
ivanshamaev/de-agent-skills/group_skills/starrocks_group_skills/starrocks_admin_cluster_health Skill
no license15★ repoStarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (information_schema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics
ivanshamaev/de-agent-skills/group_skills/trino_group_skills/trino_admin_cluster_health Skill
no license15★ repoTrino cluster health monitoring and administration — coordinator/worker health checks, REST API status endpoints (/v1/info /v1/node /v1/query /v1/cluster), JMX MBean metrics (running queries/failed queries/OOM kills/memory pool), Prometheus JMX exporter config, Grafana dashboards, query queue depth monitoring, worker node stability, memory pressure detection, BLOCKED query diagnosis, Web UI interpretation, log analysis, graceful shutdown
ivanshamaev/de-agent-skills/group_skills/trino_group_skills/trino_docker_compose_stack Skill
no license15★ repoFull local Trino Lakehouse Docker Compose stack — trinodb/trino coordinator+worker, Hive Metastore 3.x with PostgreSQL backend, MinIO object storage (S3-compatible), Apache Airflow, dbt-trino, Apache Superset BI, Prometheus+Grafana monitoring, healthchecks, startup ordering, persistent volumes, isolated networks, catalog property files, Iceberg catalog config, environment variable injection, Trino CLI access
Trino modern data stack reference architecture
ivanshamaev/de-agent-skills/group_skills/trino_group_skills/trino_modern_data_stack_reference_architecture Skill
no license15★ repoTrino Modern Data Stack end-to-end reference architecture — full docker-compose (Kafka + MinIO + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus + Grafana), medallion lake DDL and pipeline DAGs, Kafka→Iceberg streaming ingest, dbt project layout with Trino profiles, Superset dataset config, Prometheus alert rules, production Kubernetes Helm values, operational runbook for daily maintenance
ivanshamaev/de-agent-skills/group_skills/trino_group_skills/trino_observability_platform Skill
no license15★ repoTrino observability and monitoring platform — JMX Prometheus exporter configuration (running queries/failed queries/OOM kills/execution latency P50/P90/P99/memory pool metrics), Grafana dashboard panels, OpenTelemetry trace propagation, query-level event listener for structured logging, Prometheus alert rules (worker loss/queue depth/OOM/failure rate/p99 latency), log aggregation patterns, query history analysis via REST API, slow query detection SQL
Trino production readiness review
ivanshamaev/de-agent-skills/group_skills/trino_group_skills/trino_production_readiness_review Skill
no license15★ repoTrino production readiness checklist and review — coordinator HA (active/passive with load balancer), worker autoscaling, TLS/HTTPS configuration, authentication setup, resource groups for multi-tenancy, JVM sizing, spill configuration, fault-tolerant execution for batch workloads, monitoring (Prometheus/alerting), graceful shutdown, backup strategy for Hive Metastore, catalog security, query history retention, log rotation, Kubernetes deployment checklist
ivanshamaev/de-agent-skills/skills/airbyte Skill
no license15★ repoAirbyte ELT — source/destination connectors, sync modes (Full Refresh Overwrite/Append, Incremental Append/Deduped), cursor fields, primary keys, catalog/streams, deployment (abctl, Kubernetes Helm, Airbyte Cloud), Connector Builder, Python CDK (HttpStream, IncrementalMixin), normalization (dbt-based, _airbyte_raw_ tables), Airbyte API, Terraform provider, schema evolution, monitoring, Airflow AirbyteTriggerSyncOperator integration
ivanshamaev/de-agent-skills/skills/data_contracts Skill
no license15★ repoData Contracts — datacontract.com YAML specification (schema, quality, SLA, ownership, servers, changelog), Data Contract CLI (init/test/lint/diff/breaking/publish/export), SodaCL-style quality checks embedded in contract, breaking change detection (column removal/type change/constraint tightening), CI/CD GitHub Actions PR enforcement, Kafka/S3/database server definitions, contract-driven development (producer-first), semver versioning and deprecation, DataHub and OpenMetadata catalog publishing
ivanshamaev/de-agent-skills/skills/datahub_catalog Skill
no license15★ repoDataHub data catalog — metadata ingestion recipes (PostgreSQL/Hive/Spark/dbt/Airflow/Kafka/S3), Python SDK (DatahubRestEmitter, MetadataChangeProposalWrapper), column-level lineage (FineGrainedLineage), GMS/MCE/MCP architecture, Kubernetes Helm deployment, CLI operations, search/discovery REST API and GraphQL
ivanshamaev/de-agent-skills/skills/de_postmortem Skill
no license15★ repoDE blameless postmortem — incident timeline reconstruction, impact quantification, root cause chain (5 Whys), corrective actions (CAPA), postmortem document template, severity classification, SLA breach reporting, communication templates, lessons-learned facilitation, postmortem review checklist
ivanshamaev/de-agent-skills/skills/de_production_readiness Skill
no license15★ repoData engineering production readiness — idempotency patterns (INSERT ON CONFLICT, MERGE, partition overwrite), retry strategies (exponential backoff, Airflow retry config), SLA monitoring (Airflow sla_miss_callback, Prometheus PromQL), data freshness checks (Soda freshness, max event_time), structured logging (JSON logs, OpenTelemetry tracing), data reconciliation (row count, checksum, duplicate detection), graceful degradation (circuit breaker, fallback, skip failed partitions), change management (blue/green table swap, schema migration rollback)
ivanshamaev/de-agent-skills/skills/docker_data_envs Skill
no license15★ repoDocker for data engineering environments — multi-stage Dockerfiles for dbt/Spark/Airflow images, BuildKit layer caching (--mount=type=cache), private registries (ghcr.io/Harbor), docker buildx multi-platform builds, Docker Compose local data stacks (Spark+Airflow+Kafka+MinIO+Postgres), BuildKit secrets for private PyPI, security hardening (non-root user, slim base images, read-only FS), CI/CD with GitHub Actions build-push-action
ivanshamaev/de-agent-skills/skills/github_actions_dataops Skill
no license15★ repoUse when building or reviewing GitHub Actions CI/CD pipelines for data engineering — covering dbt slim CI (state:modified+, --defer, manifest.json), SQLFluff SQL linting with PR annotations, Airflow DAG integrity tests (pytest/DagBag), Great Expectations and Soda data quality gates, multi-stage Docker image builds for Spark/dbt/Airflow pushed to ghcr.io, secrets management with OIDC for AWS/GCP (no static keys), environment-scoped secrets, reusable workflows (workflow_call), matrix builds, composite actions, and full end-to-end dbt project CI/CD workflows.
ivanshamaev/de-agent-skills/skills/kubernetes_data Skill
no license15★ repoData platform on Kubernetes — Spark-on-K8s (spark-submit, pod templates, dynamic allocation, RBAC), Airflow on K8s (Helm chart, KubernetesExecutor, KubernetesPodOperator, git-sync, DAG storage), resource management, namespaces, autoscaling
ivanshamaev/de-agent-skills/skills/mage_ai Skill
no license15★ repoMage AI data pipelines — block types (data loader/transformer/exporter/sensor/custom), pipeline YAML, hybrid SQL+Python blocks, triggers (schedule/event/API), streaming pipelines, dbt integration, Spark integration, Docker/Kubernetes deployment, io_config.yaml, pipeline variables, callbacks, backfills
ivanshamaev/de-agent-skills/skills/mcp_server Skill
no license15★ repoMCP (Model Context Protocol) server development — FastMCP/Python SDK, tools/resources/prompts primitives, STDIO and Streamable HTTP transports, Claude Desktop/Claude Code client config, MCP Inspector testing, security best practices (input validation/OAuth2/confused deputy prevention), production Docker/Kubernetes deployment, agentic data platform integration
ivanshamaev/de-agent-skills/skills/prefect_workflows Skill
no license15★ repoPrefect 3.x workflows — flows, tasks, deployments, work pools, event-driven triggers, caching, retries, state hooks, Prefect Cloud/server, Python SDK, Docker/K8s infrastructure
ivanshamaev/de-agent-skills/skills/redpanda Skill
no license15★ repoRedpanda Kafka-compatible streaming — cluster setup, topic config, rpk CLI, producer/consumer tuning, tiered storage, Schema Registry, Kafka Connect compatibility, monitoring, Docker/Kubernetes deployment
ivanshamaev/de-agent-skills/skills/sqlfluff Skill
no license15★ repoSQLFluff SQL linter — .sqlfluff config, dialect selection (ansi/bigquery/clickhouse/duckdb/hive/postgres/snowflake/sparksql/trino), rule sets, templating (Jinja/dbt), fix command, VS Code integration, pre-commit hook, CI/CD GitHub Actions, custom rules, noqa inline suppression
ivanshamaev/de-agent-skills/skills/terraform_data Skill
no license15★ repoTerraform for data infrastructure — S3/MinIO data lake buckets (versioning, lifecycle, SSE-KMS), IAM roles for Spark/Airflow (least-privilege, IRSA on EKS), MSK/Kafka clusters (aws_msk_cluster, encryption, custom broker config), Kubernetes data platform (helm_release Airflow + Spark, namespace resource quotas), module layout (modules/ + envs/), typed variables with validation, S3 remote state + DynamoDB locking, Terragrunt DRY configs, GitHub Actions CI/CD pipeline with plan/apply
byronxlg/skillfold/library/skills/github-workflow Skill
15★ repoWork with GitHub branches, PRs, issues, and reviews via the gh CLI.
HetCreep/CoalMine/plugin/skills/telemetry-canary Skill
15★ repoNine quality-canary skills for AI coding agents - code health, world-class rule completeness, grounding, supply chain, resilience, drift & more - plus the auto-cadence hooks that run them unprompted at session start and session end. Cross-agent, consent-gated, token-lean.
HetCreep/CoalMine/skills/telemetry-canary Skill
15★ repoObservability and structured logging canary — checks for structured logs (JSON), OpenTelemetry metrics/traces, proper error stack traces, and flags empty catches or silent log swallowing. Triggers on keywords: "/telemetry-canary", "telemetry-canary", "observability audit", "structured logging". Use when adding or changing logging, metrics, tracing, or error-handling code.
G1Joshi/Agent-Skills/skills/architecture/serverless Skill
15★ repoServerless architecture with FaaS and BaaS. Use for cloud functions.
G1Joshi/Agent-Skills/skills/databases/dynamodb Skill
15★ repoAWS DynamoDB serverless NoSQL with streams and global tables. Use for AWS serverless.
G1Joshi/Agent-Skills/skills/devops/argocd Skill
15★ repoArgoCD GitOps continuous delivery for Kubernetes. Use for K8s GitOps.
G1Joshi/Agent-Skills/skills/devops/aws Skill
15★ repoAmazon Web Services cloud platform with Lambda, EC2, S3, and RDS. Use for AWS infrastructure.
G1Joshi/Agent-Skills/skills/devops/azure Skill
15★ repoMicrosoft Azure cloud platform with Functions, AKS, and DevOps. Use for Azure services.
G1Joshi/Agent-Skills/skills/devops/circleci Skill
15★ repoCircleCI cloud CI/CD platform. Use for continuous integration.
G1Joshi/Agent-Skills/skills/devops/cloudflare Skill
15★ repoCloudflare CDN, Workers, and edge services. Use for CDN and edge.
G1Joshi/Agent-Skills/skills/devops/datadog Skill
15★ repoDatadog monitoring and APM platform. Use for observability.
G1Joshi/Agent-Skills/skills/devops/docker Skill
15★ repoDocker containerization with Dockerfile, Compose, and multi-stage builds. Use for containers.
G1Joshi/Agent-Skills/skills/devops/gcp Skill
15★ repoGoogle Cloud Platform with Cloud Run, GKE, and BigQuery. Use for GCP infrastructure.
G1Joshi/Agent-Skills/skills/devops/github-actions Skill
15★ repoGitHub Actions CI/CD workflows with reusable actions. Use for GitHub automation.
G1Joshi/Agent-Skills/skills/devops/gitlab-ci Skill
15★ repoGitLab CI/CD pipelines with runners and stages. Use for GitLab automation.
G1Joshi/Agent-Skills/skills/devops/grafana Skill
15★ repoGrafana dashboards and visualization for metrics. Use for observability.
G1Joshi/Agent-Skills/skills/devops/helm Skill
15★ repoHelm Kubernetes package manager with charts. Use for K8s deployments.
G1Joshi/Agent-Skills/skills/devops/jenkins Skill
15★ repoJenkins automation server with pipelines and plugins. Use for CI/CD pipelines.
G1Joshi/Agent-Skills/skills/devops/kubernetes Skill
15★ repoKubernetes container orchestration with Helm, operators, and service mesh. Use for cluster management.
G1Joshi/Agent-Skills/skills/devops/openshift Skill
15★ repoOpenShift enterprise Kubernetes platform. Use for enterprise K8s.
G1Joshi/Agent-Skills/skills/devops/prometheus Skill
15★ repoPrometheus monitoring and alerting with PromQL. Use for metrics collection.
G1Joshi/Agent-Skills/skills/devops/rancher Skill
15★ repoRancher Kubernetes management platform. Use for multi-cluster K8s.
G1Joshi/Agent-Skills/skills/devops/terraform Skill
15★ repoTerraform infrastructure as code with providers and state management. Use for cloud provisioning.
G1Joshi/Agent-Skills/skills/devops/vercel Skill
15★ repoVercel frontend deployment with edge functions. Use for frontend hosting.
G1Joshi/Agent-Skills/skills/tools/docker-desktop Skill
15★ repoDocker Desktop for Mac/Windows. Use for local containers.
G1Joshi/Agent-Skills/skills/tools/k9s Skill
15★ repok9s terminal UI for Kubernetes. Use for K8s management.