Cicd pipelines

CI/CD Pipelines - Comprehensive DevOps Skill

A unified skill for CI/CD pipeline design, DevOps automation, infrastructure as code, GitOps deployment automation, security scanning, and enterprise pipeline readiness across major platforms.

When to Use This Skill

Use this skill when:

CI/CD Pipeline Development

• Creating new CI/CD workflows (GitHub Actions, GitLab CI, Jenkins)
• Debugging pipeline failures or flaky tests
• Optimizing slow builds or test suites
• Implementing caching strategies
• Setting up deployment workflows
• Implementing matrix builds or test sharding
• Configuring multi-environment deployments

DevSecOps & Security

• Securing pipelines (secrets, OIDC, supply chain)
• Implementing security scanning (SAST, DAST, SCA)
• Container image vulnerability scanning inside a pipeline
• Secret detection and management
• Enterprise readiness assessment (OpenSSF compliance)

Infrastructure as Code

• Terraform module development
• CloudFormation/CDK templates
• Deploying Kubernetes manifests and Helm charts from CI/CD
• GitOps workflows (ArgoCD, Flux)

Container Pipeline Integration

• Building and signing container images from an existing Dockerfile
• Publishing to registries from CI/CD
• Scanning container images with tools such as Trivy or Snyk
• Deploying containerized services through GitOps

For Dockerfile design, Docker Compose, local container environments, or container runtime architecture, switch to docker-containerization.

Release Management

• Semantic versioning automation
• Changelog generation
• GitHub/GitLab release creation
• Artifact signing and provenance

Quick Start

1. Creating a New Pipeline

Decision tree:

What are you building?
+-- Node.js/Frontend --> templates/github-actions/node-ci.yml | templates/gitlab-ci/node-ci.yml
+-- Python --> templates/github-actions/python-ci.yml | templates/gitlab-ci/python-ci.yml
+-- Go --> templates/github-actions/go-ci.yml | templates/gitlab-ci/go-ci.yml
+-- Container image pipeline --> templates/github-actions/docker-build.yml | templates/gitlab-ci/docker-build.yml
+-- Security Scanning --> templates/github-actions/security-scan.yml | templates/gitlab-ci/security-scan.yml

Basic pipeline structure:

# 1. Fast feedback (lint, format) - <1 min
# 2. Unit tests - 1-5 min
# 3. Integration tests - 5-15 min
# 4. Build artifacts
# 5. E2E tests (optional, main branch only) - 15-30 min
# 6. Deploy (with approval gates)

2. Optimizing Pipeline Performance

Quick wins checklist:

• Add dependency caching (50-90% faster builds)
• Remove unnecessary needs dependencies
• Add path filters to skip unnecessary runs
• Use npm ci instead of npm install
• Add job timeouts to prevent hung builds
• Enable concurrency cancellation for duplicate runs

Analyze existing pipeline:

# Use the pipeline analyzer script
python3 scripts/pipeline_analyzer.py --platform github --workflow .github/workflows/ci.yml

3. Securing Your Pipeline

Essential security checklist:

• Use OIDC instead of static credentials
• Pin actions/includes to commit SHAs
• Use minimal permissions
• Enable secret scanning
• Add vulnerability scanning (dependencies, containers)
• Implement branch protection
• Separate test from deploy workflows

OIDC Authentication (GitHub Actions to AWS):

permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789:role/GitHubActionsRole
      aws-region: us-east-1

Core Capabilities

Domain	Tools & Technologies
CI/CD Platforms	GitHub Actions, GitLab CI, Jenkins
Infrastructure as Code	Terraform, AWS CDK, CloudFormation, Pulumi
Container Pipeline Integration	Image build/publish steps, Trivy/Snyk scanning, Cosign signing, Kubernetes/Helm deploy jobs
GitOps	ArgoCD, Flux
Security Scanning	CodeQL, Semgrep, Trivy, Snyk, TruffleHog
Cloud Platforms	AWS, Azure, GCP, Cloudflare
Release Artifacts & Provenance	Cosign, SLSA, signed tags, reproducible build checks

Architecture Patterns

CI/CD Pipeline Flow

Code Commit --> Build --> Test --> Security Scan --> Package
                                                       |
Monitor <-- Release Staging <-- Smoke Tests <-- Deploy Dev
                 |
         Manual Approval
                 |
         Deploy Production

GitOps Architecture

App Repo --CI--> Config Repo --ArgoCD--> K8s Cluster
                     ^                        |
                     +----Continuous Sync-----+

Reference Documentation

CI/CD & Pipeline Design

• references/best_practices.md - Pipeline design patterns, testing strategies, deployment workflows
• references/optimization.md - Caching strategies, parallelization, build performance
• references/troubleshooting.md - Common issues, debugging, platform-specific problems
• references/cicd-github-actions.md - GitHub Actions workflows, runners, secrets

Security & DevSecOps

• references/security.md - Secrets management, OIDC, supply chain security
• references/devsecops.md - SAST, DAST, SCA, container scanning guide
• references/devsecops-basics.md - Security best practices, shift-left security
• references/security-hardening.md - TLS enforcement, input validation, headers

Infrastructure & Cloud Platforms

• references/terraform-eks-module.tf - Production EKS cluster Terraform
• references/kubernetes-deployment.yaml - Example manifest deployed by pipelines
• references/kubernetes-basics.md - Kubernetes concepts needed to understand deployment jobs
• references/docker-basics.md - Pipeline-adjacent container concepts; use docker-containerization for Dockerfile design
• references/docker-compose.md - Pipeline-adjacent compose references; use docker-containerization for local container environments
• references/aws-overview.md - AWS fundamentals, IAM, services
• references/gcloud-platform.md - GCP overview, gcloud CLI
• references/cloudflare-workers-basics.md - Edge computing, Workers

Enterprise Readiness

• references/general.md - Universal enterprise readiness checks
• references/github.md - GitHub-specific enterprise requirements
• references/openssf-badge-silver.md - Silver badge criteria
• references/openssf-badge-gold.md - Gold badge criteria
• references/signed-releases.md - Artifact and tag signing
• references/reproducible-builds.md - Deterministic build patterns

Release Artifact References

• references/signed-releases.md - Artifact and tag signing
• references/reproducible-builds.md - Deterministic build patterns

Templates

GitHub Actions

Template	Description
templates/github-actions/node-ci.yml	Complete Node.js CI/CD with security scanning
templates/github-actions/python-ci.yml	Python pipeline with pytest, coverage, PyPI
templates/github-actions/go-ci.yml	Go pipeline with multi-platform builds
templates/github-actions/docker-build.yml	Docker build with multi-platform, signing
templates/github-actions/security-scan.yml	Comprehensive DevSecOps pipeline
templates/github-actions/dco-check.yml	DCO sign-off enforcement

GitLab CI

Template	Description
templates/gitlab-ci/node-ci.yml	GitLab CI Node.js pipeline
templates/gitlab-ci/python-ci.yml	Python pipeline with parallel testing
templates/gitlab-ci/go-ci.yml	Go pipeline with Kubernetes deployment
templates/gitlab-ci/docker-build.yml	Docker build with DinD, multi-arch
templates/gitlab-ci/security-scan.yml	DevSecOps with GitLab security templates

Enterprise Templates

Template	Description
templates/GOVERNANCE.md	Project governance documentation
templates/ARCHITECTURE.md	Technical architecture template
templates/CODE_OF_CONDUCT.md	Contributor Covenant v2.1
templates/SECURITY_AUDIT.md	Security self-audit template

Scripts

Pipeline Analysis

Script	Description
scripts/pipeline_analyzer.py	Analyze workflows for optimization opportunities
scripts/ci_health.py	Check pipeline status and identify issues
scripts/validate-devops-skill.sh	Validate DevOps configurations

Security & Compliance

Script	Description
scripts/verify-badge-criteria.sh	OpenSSF Badge verification
scripts/check-coverage-threshold.sh	Statement coverage validation
scripts/check-branch-coverage.sh	Branch coverage analysis
scripts/verify-signed-tags.sh	Git tag signature verification
scripts/verify-review-requirements.sh	PR review requirements check
scripts/check-tls-minimum.sh	TLS 1.2+ enforcement check
scripts/verify-spdx-headers.sh	SPDX license header verification
scripts/add-spdx-headers.sh	Add SPDX headers to files

Infrastructure

Script	Description
scripts/cloudflare_deploy.py	Cloudflare Worker deployments
scripts/docker_optimize.py	Dockerfile analysis and optimization

Anti-Patterns

Anti-Pattern	Symptom	Fix
YAML copy-paste proliferation	Identical workflows duplicated across repos	Reusable workflows, Helm charts, Kustomize bases, Terraform modules
Hardcoded secrets in code	API keys/passwords committed to git	Secret managers (Vault, AWS SM), sealed secrets, env vars from secure sources
No rollback strategy	No plan for deployment failure	Blue/green, canary with automated rollback, ArgoCD auto-revert
Monolithic CI pipeline	Single 45-minute pipeline on every commit	Parallel jobs, caching, incremental builds, path-based triggers
Running as root in containers	No USER instruction, privileged pods	Add USER instruction, set securityContext.runAsNonRoot: true
Using :latest tags	FROM node:latest in production	Pin specific versions, use immutable tags with SHA digests
Script injection vulnerability	${{ github.event.* }} directly in run: blocks	Use environment variables instead (see below)
Missing resource limits	Pods consume unbounded resources	Set requests and limits for CPU/memory in all deployments
Unpinned GitHub Actions	uses: actions/checkout@v4 without SHA	Pin to commit SHA: uses: actions/checkout@b4ffde6

Script injection fix:

# DANGEROUS
- run: echo "Title: ${{ github.event.issue.title }}"

# SAFE
- name: Process issue
  env:
    TITLE: ${{ github.event.issue.title }}
  run: echo "Title: $TITLE"

Quick Reference Commands

GitHub Actions

gh workflow list                    # List workflows
gh run list --limit 20              # View recent runs
gh run view <run-id>                # View specific run
gh run rerun <run-id> --failed      # Re-run failed jobs
gh run view <run-id> --log > logs.txt  # Download logs
gh workflow run ci.yml              # Trigger workflow manually

GitLab CI

gl project-pipelines list           # View pipelines
gl project-pipeline get <id>        # Pipeline status
gl project-pipeline retry <id>      # Retry failed jobs
gl project-pipeline cancel <id>     # Cancel pipeline

Docker

docker build -t myapp .             # Build image
docker run -p 3000:3000 myapp       # Run container
docker compose up -d                # Start multi-container app
docker scout cves myapp             # Scan for vulnerabilities

Kubernetes

kubectl apply -f deployment.yaml    # Apply manifest
kubectl get pods,services           # Check status
kubectl logs -f <pod>               # Stream logs
kubectl rollout status deployment/app  # Check rollout

Terraform

terraform init                      # Initialize
terraform plan                      # Preview changes
terraform apply                     # Apply changes
terraform state list                # List resources

Quality Checklist

[ ] All secrets in secret management (not in code)
[ ] Resource limits defined for all containers
[ ] Health checks configured (liveness, readiness)
[ ] Horizontal pod autoscaling enabled
[ ] Security contexts set (non-root, read-only)
[ ] Monitoring and alerting configured
[ ] Rollback strategy documented
[ ] Multi-environment support (dev, staging, prod)
[ ] Concurrency controls in CI pipelines
[ ] Remote state backend for Terraform
[ ] Vulnerability scanning in pipeline
[ ] Version pinning for all dependencies
[ ] Branch protection enabled
[ ] Code review required before merge

Platform Selection Guide

Need	Choose
Sub-50ms latency globally	Cloudflare Workers
Serverless functions (AWS)	AWS Lambda
Containerized workloads	AWS ECS/Fargate, GKE, AKS
Kubernetes at scale	AWS EKS, Azure AKS, GCP GKE
Object storage (zero egress)	Cloudflare R2
Managed SQL	AWS RDS, Azure SQL, Cloud SQL
GitHub-integrated CI/CD	GitHub Actions
Self-hosted CI/CD	GitLab CI, Jenkins
Kubernetes GitOps	ArgoCD, Flux
Predictable workloads	Reserved Instances, Savings Plans
Fault-tolerant workloads	Spot Instances, Preemptible VMs

Getting Started

1. New pipeline: Start with a template from templates/
2. Add security scanning: Use DevSecOps templates or add security stages
3. Optimize existing: Run scripts/pipeline_analyzer.py
4. Debug issues: Check references/troubleshooting.md
5. Improve security: Review references/security.md and references/devsecops.md
6. Enterprise readiness: Follow references/general.md checklist
7. Release notes/versioning: switch to git-workflow for changelog, semantic version, and commit convention workflows

Source Skills

This curated skill combines content from the following legacy skills (now part of cicd-pipelines):

• Pipeline design, DevSecOps, optimization
• IaC, Kubernetes, deployment automation
• Cloud platforms, Docker, Cloudflare
• Multi-cloud, FinOps, comprehensive DevOps
• OpenSSF compliance, security assessment
• Artifact signing, provenance, and pipeline gates

Resources

• GitHub Actions: https://docs.github.com/actions
• GitLab CI: https://docs.gitlab.com/ee/ci/
• Terraform: https://developer.hashicorp.com/terraform
• Kubernetes: https://kubernetes.io/docs
• ArgoCD: https://argo-cd.readthedocs.io
• OpenSSF Scorecard: https://securityscorecards.dev/
• SLSA Framework: https://slsa.dev/