Ci cd pipeline engineering

CI/CD Pipeline Engineering

Overview

The pipeline is the enforcement mechanism for every other practice. Code review, scanning, testing, signing, promotion — if it's not in the pipeline, it's optional, and optional doesn't happen under pressure.

Fail fast, shift left. Cheap, fast checks run first. Formatting and linting in seconds, unit tests in a minute, builds after that, expensive security scans and integration tests later. Never run a 20-minute suite to discover a syntax error.

Build once, promote everywhere. The artifact tested in staging is bit-for-bit the artifact deployed to production — same digest, no rebuilds. Each environment rebuild is a different, untested artifact.

Pipelines are production infrastructure. They hold credentials, produce artifacts, and touch production. They get the same rigor: review, least privilege, audit logging.

When to Use

• Creating a new pipeline (GitHub Actions, GitLab CI/CD, Jenkins)
• Adding quality gates or security scanning to an existing pipeline
• Debugging CI failures
• Setting up OIDC authentication from pipelines to cloud providers
• Designing artifact promotion across environments
• Configuring environments, approvals, and protected deployments

The Canonical Pipeline Shape

Pull Request                          Merge to main
    │                                     │
    ▼                                     ▼
┌─────────────────┐               ┌──────────────────┐
│ 1. FORMAT+LINT  │ (seconds)     │ All PR gates     │
│ 2. TYPECHECK    │               │      +           │
│ 3. UNIT TESTS   │ (1-2 min)     │ 5. BUILD ARTIFACT│
│ 4. STATIC SCAN  │ (secrets,SAST)│ 6. IMAGE SCAN    │
└─────────────────┘               │ 7. SIGN + SBOM   │
        │                         │ 8. PUSH REGISTRY │
        ▼                         └──────────────────┘
   Review & merge                          │
                                           ▼
                              ┌──────────────────────────┐
                              │ DEPLOY staging (auto)    │
                              │ smoke tests              │
                              │ DEPLOY prod (approval)   │
                              │ verify: health + metrics │
                              └──────────────────────────┘

No gate is skippable. A red gate blocks the pipeline. The fix is in the code, not in the gate's configuration.

GitHub Actions: Reference Workflow

name: ci

on:
  pull_request:
  push:
    branches: [main]

# Least privilege by default; elevate per-job only when needed
permissions: {}

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lint-and-test:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@<full-commit-sha>   # v4 — pin third-party actions by SHA
      - uses: actions/setup-python@<full-commit-sha>
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements-dev.txt
      - run: ruff check .
      - run: pytest --cov

  secrets-scan:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@<full-commit-sha>
      - uses: gitleaks/gitleaks-action@<full-commit-sha>

  build-and-scan:
    needs: [lint-and-test, secrets-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      packages: write          # push to GHCR
      id-token: write          # OIDC for cosign keyless signing
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@<full-commit-sha>
      - uses: docker/setup-buildx-action@<full-commit-sha>
      - uses: docker/login-action@<full-commit-sha>
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - id: build
        uses: docker/build-push-action@<full-commit-sha>
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - run: |
          trivy image --severity HIGH,CRITICAL --exit-code 1 \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}
      # SBOM + provenance: see supply-chain-security skill
      - run: syft ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }} -o spdx-json > sbom.json
      - uses: actions/upload-artifact@<full-commit-sha>
        with: { name: sbom, path: sbom.json }

  deploy-staging:
    needs: build-and-scan
    runs-on: ubuntu-24.04
    environment: staging
    permissions:
      id-token: write    # OIDC: no long-lived cloud keys
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@<full-commit-sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-deploy-staging
          aws-region: eu-west-1
      - run: ./scripts/deploy.sh staging ghcr.io/${{ github.repository }}@${{ needs.build-and-scan.outputs.digest }}
      - run: ./scripts/smoke-test.sh staging

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-24.04
    environment: production    # requires manual approval via environment protection rules
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@<full-commit-sha>
        with:
          role-to-assume: arn:aws:iam::<acct>:role/gha-deploy-prod
          aws-region: eu-west-1
      - run: ./scripts/deploy.sh production ghcr.io/${{ github.repository }}@${{ needs.build-and-scan.outputs.digest }}
      - run: ./scripts/verify-deployment.sh production

Note the digest: staging and production deploy ${{ needs.build-and-scan.outputs.digest }} — the exact image scanned, not a re-resolution of a mutable tag. That is build-once-promote-everywhere made concrete.

OIDC Instead of Long-Lived Cloud Keys

Static cloud credentials in CI secrets are a top-tier risk: they leak in logs, live forever, and get copied to places you can't audit. OIDC replaces them:

GitHub Actions ──(signed OIDC token: repo, ref, workflow)──▶ Cloud STS
                                                              │
                                                              ▼
                                            Short-lived credentials (minutes),
                                            scoped to an IAM role with a trust
                                            policy pinning repo + branch + env

• AWS: aws-actions/configure-aws-credentials + an IAM OIDC provider and role trust policy on repo:org/repo:environment:production.
• Azure: azure/login with federated credentials on a service principal / managed identity.
• GCP: google-github-actions/auth with Workload Identity Federation.

The trust policy must scope to repo + environment/branch, never repo:org/repo:* for production roles.

Pipeline Security Rules (non-negotiable)

1. permissions: {} at workflow level — every job declares exactly what it needs.
2. Pin third-party actions by full commit SHA, not by tag (@v4 is mutable). Review what you pin.
3. Never run untrusted code with secrets. pull_request_target checking out fork code is remote-code-execution-as-a-service. Fork PRs get no secrets, period.
4. Environment protection rules on production: required reviewers, and deployment branches restricted to main.
5. No self-hosted runners on public repos — fork PRs would execute on your infrastructure.
6. Secrets masked and minimal. CI never has production secrets it doesn't need; staging and production use different identities.
7. Step output is untrusted — never run: echo "${{ github.event.issue.title }}" unquoted into a shell (script injection). Pass through env vars instead.

Pipeline Speed Discipline

Pipelines over ~10 minutes get bypassed by humans. Optimize in this order:

1. Cache dependencies (cache: pip/npm, type=gha for Docker layers).
2. Parallelize independent jobs — lint, unit tests, and secret scans don't depend on each other.
3. Fail fast ordering — cheap gates before expensive ones (the shape above).
4. Path filters — docs-only PRs skip the image build.
5. Matrix sharding for large test suites.
6. Right-size runners — bigger runners or self-hosted for heavy builds.

GitLab CI / Jenkins Notes

• GitLab CI/CD: same shape — stages: with early lint/test, rules:/workflow:rules: for control, id_token + aud for OIDC to cloud, protected environments for approvals, needs: for DAG parallelization.
• Jenkins: learn it because it's everywhere; don't choose it for greenfield without a reason. Declarative Jenkinsfile in the repo, agents as containers, credentials in a real secret store (Vault/Kubernetes secrets), and controller hardening owned by the platform team.

CI Failure Triage

CI red?
 ├── Read the ACTUAL failure, not the last line — scroll to the first ERROR
 ├── Reproduce locally (same command, same container if possible)
 ├── Classify:
 │    ├── Deterministic code failure → fix the code
 │    ├── Flaky test → fix the flake (quarantine is a last resort with an expiry date)
 │    ├── Infra failure (runner, network, registry) → retry once, then investigate the platform
 │    └── Dependency drift (new upstream version) → pin and fix deliberately
 └── Never merge with red CI "to fix forward" — main stays green or the pipeline loses all authority

Common Rationalizations

Rationalization	Reality
"The pipeline is slow, let's drop a gate"	Optimize the pipeline (cache, parallelism), don't remove the protection. A skipped scan ships to production eventually.
"Pinning actions by SHA makes updates painful"	That pain is called review. Renovate/Dependabot can bump SHAs with PRs. Mutable tags mean a compromised action silently becomes your deploy pipeline.
"OIDC is complex; a stored key works"	The stored key will leak, will be over-privileged, and will still be valid in 5 years on an ex-employee's laptop. OIDC is a one-time setup.
"We rebuild per environment to inject config"	Rebuilding per environment means production runs an artifact that never passed your tests. Inject config at runtime; promote the digest.
"Approvals on every step keep us safe"	Approval fatigue means rubber-stamping. Gate what matters: production, destructive changes, DB migrations, critical infra. Automate the rest.
"main is broken but my PR is unrelated, force merge"	A red main hides every subsequent failure. Fix or revert first — the Build Cop owns this.

Red Flags

• Secrets stored as plain CI variables instead of masked secrets, or worse, in the YAML
• permissions: write-all / no permissions block at all
• Third-party actions referenced by mutable tags (@v1, @main)
• pull_request_target with a checkout of fork code
• Production deploys that skip staging
• Pipeline if: always() hacks around failing gates
• One 45-minute sequential pipeline nobody wants to touch
• Deploy steps that kubectl apply / ssh from a generic CI job with cluster-admin

Verification

• Every PR runs lint, typecheck, unit tests, and secret scanning in under ~5 minutes
• Full pipeline (build, scan, sign, push) runs on merge to main
• Workflow-level permissions: {}; per-job minimal permissions
• All third-party actions pinned by full commit SHA
• Cloud authentication via OIDC; zero long-lived cloud keys in secrets
• Staging deploys automatically; production requires approval and deploys the same digest
• A failed gate blocks merge via branch protection
• Pipeline failures have a documented triage path; flaky tests have owners and expiry dates
• Deployment traceability: from any deployed artifact you can find commit → pipeline run → approver → environment → timestamp