Deployment patterns

Deployment Patterns

Production deployment workflows and CI/CD best practices.

---

When to Activate

• Setting up CI/CD pipelines
• Dockerizing an application
• Planning deployment strategy (blue-green, canary, rolling)
• Implementing health checks and readiness probes
• Preparing for a production release
• Configuring environment-specific settings

---

Deployment Strategies

Rolling Deployment (Default)

Replace instances gradually, old and new versions run simultaneously during rollout.

Instance 1: v1 → v2  (update first)
Instance 2: v1        (still running v1)
Instance 3: v1        (still running v1)

Instance 1: v2
Instance 2: v1 → v2  (update second)
Instance 3: v1

Instance 1: v2
Instance 2: v2
Instance 3: v1 → v2  (update last)

Pros: Zero downtime, gradual rollout Cons: Two versions run simultaneously, requires backward-compatible changes Use when: Standard deployments, backward-compatible changes

Blue-Green Deployment

Run two identical environments. Switch traffic atomically.

Blue  (v1) ← traffic
Green (v2)   idle, running new version

# After verification:
Blue  (v1)   idle (becomes standby)
Green (v2) ← traffic

Pros: Instant rollback (switch back to blue), clean cutover Cons: Requires 2x infrastructure during deployment Use when: Critical services, zero-tolerance for issues

Canary Deployment

Route a small percentage of traffic to the new version first.

v1: 95% of traffic
v2:  5% of traffic  (canary)

# If metrics look good:
v1: 50% of traffic
v2: 50% of traffic

# Final:
v2: 100% of traffic

Pros: Catches issues with real traffic before full rollout Cons: Requires traffic splitting infrastructure, monitoring Use when: High-traffic services, risky changes, feature flags

---

Docker

Multi-Stage Dockerfile (Node.js)

# Stage 1: Install dependencies
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --production=false

# Stage 2: Build
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
RUN npm prune --production

# Stage 3: Production image
FROM node:22-alpine AS runner
WORKDIR /app

RUN addgroup -g 1001 -S appgroup && adduser -S appuser -u 1001
USER appuser

COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /app/dist ./dist
COPY --from=builder --chown=appuser:appgroup /app/package.json ./

ENV NODE_ENV=production
EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

CMD ["node", "dist/server.js"]

Multi-Stage Dockerfile (Go)

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /server ./cmd/server

FROM alpine:3.19 AS runner
RUN apk --no-cache add ca-certificates
RUN adduser -D -u 1001 appuser
USER appuser

COPY --from=builder /server /server

EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s CMD wget -qO- http://localhost:8080/health || exit 1
CMD ["/server"]

Multi-Stage Dockerfile (Python/Django)

FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir uv
COPY requirements.txt .
RUN uv pip install --system --no-cache -r requirements.txt

FROM python:3.12-slim AS runner
WORKDIR /app

RUN useradd -r -u 1001 appuser
USER appuser

COPY --from=builder /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY . .

ENV PYTHONUNBUFFERED=1
EXPOSE 8000

HEALTHCHECK --interval=30s --timeout=3s CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health/')" || exit 1
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "4"]

Docker Best Practices

# GOOD practices
- Use specific version tags (node:22-alpine, not node:latest)
- Multi-stage builds to minimize image size
- Run as non-root user
- Copy dependency files first (layer caching)
- Use .dockerignore to exclude node_modules, .git, tests
- Add HEALTHCHECK instruction
- Set resource limits in docker-compose or k8s

# BAD practices
- Running as root
- Using :latest tags
- Copying entire repo in one COPY layer
- Installing dev dependencies in production image
- Storing secrets in image (use env vars or secrets manager)

---

CI/CD Pipeline

GitHub Actions (Standard Pipeline)

name: CI/CD

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test -- --coverage
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: coverage
          path: coverage/

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    # The "production" environment must have a GitHub environment protection rule
    # with required reviewers, so this job pauses for an explicit human approval
    # before it runs. Merge to main must never auto-deploy to prod without that gate.
    environment: production
    steps:
      - name: Deploy to production
        run: |
          # Platform-specific deployment command
          # Railway: railway up
          # Vercel: vercel --prod
          # K8s: kubectl set image deployment/app app=ghcr.io/${{ github.repository }}:${{ github.sha }}
          echo "Deploying ${{ github.sha }}"

Gate the production deploy behind an explicit approval. Either protect the production GitHub environment with required reviewers (as above), or move production promotion to a separate workflow_dispatch trigger that a human runs on demand. Merging to main may build the image and deploy to staging automatically, but it must not promote to production without that explicit gate.

Pipeline Stages

PR opened:
  lint → typecheck → unit tests → integration tests → preview deploy

Merged to main:
  lint → typecheck → unit tests → integration tests → build image → deploy staging → smoke tests
  → [explicit manual approval gate] → deploy production

Everything up to and including the staging smoke tests runs automatically on merge. The final deploy-production step is a separate, explicitly approved gate (an environment protection rule with required reviewers, or a manually triggered workflow), not an automatic link in the merge chain.

---

Health Checks

Health Check Endpoint

// Simple health check
app.get("/health", (req, res) => {
  res.status(200).json({ status: "ok" });
});

// Detailed health check (for internal monitoring)
app.get("/health/detailed", async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    externalApi: await checkExternalApi(),
  };

  const allHealthy = Object.values(checks).every(c => c.status === "ok");

  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? "ok" : "degraded",
    timestamp: new Date().toISOString(),
    version: process.env.APP_VERSION || "unknown",
    uptime: process.uptime(),
    checks,
  });
});

async function checkDatabase(): Promise<HealthCheck> {
  try {
    await db.query("SELECT 1");
    return { status: "ok", latency_ms: 2 };
  } catch (err) {
    return { status: "error", message: "Database unreachable" };
  }
}

Kubernetes Probes

livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 10
  periodSeconds: 30
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 2

startupProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 0
  periodSeconds: 5
  failureThreshold: 30    # 30 * 5s = 150s max startup time

---

Environment Configuration

Twelve-Factor App Pattern

# All config via environment variables — never in code
DATABASE_URL=postgres://user:pass@host:5432/db
REDIS_URL=redis://host:6379/0
API_KEY=${API_KEY}           # injected by secrets manager
LOG_LEVEL=info
PORT=3000

# Environment-specific behavior
NODE_ENV=production          # or staging, development
APP_ENV=production           # explicit app environment

Configuration Validation

import { z } from "zod";

const envSchema = z.object({
  NODE_ENV: z.enum(["development", "staging", "production"]),
  PORT: z.coerce.number().default(3000),
  DATABASE_URL: z.string().url(),
  REDIS_URL: z.string().url(),
  JWT_SECRET: z.string().min(32),
  LOG_LEVEL: z.enum(["debug", "info", "warn", "error"]).default("info"),
});

// Validate at startup — fail fast if config is wrong
export const env = envSchema.parse(process.env);

---

Rollback Strategy

Instant Rollback

# Docker/Kubernetes: point to previous image
kubectl rollout undo deployment/app

# Vercel: promote previous deployment
vercel rollback

# Railway: redeploy previous commit
railway up --commit <previous-sha>

# Database: rollback migration (if reversible)
npx prisma migrate resolve --rolled-back <migration-name>

Rollback Checklist

• Previous image/artifact is available and tagged
• Database migrations are backward-compatible (no destructive changes)
• Feature flags can disable new features without deploy
• Monitoring alerts configured for error rate spikes
• Rollback tested in staging before production release

---

Production Readiness Checklist

Before any production deployment:

Application

• All tests pass (unit, integration, E2E)
• No hardcoded secrets in code or config files
• Error handling covers all edge cases
• Logging is structured (JSON) and does not contain PII
• Health check endpoint returns meaningful status

Infrastructure

• Docker image builds reproducibly (pinned versions)
• Environment variables documented and validated at startup
• Resource limits set (CPU, memory)
• Horizontal scaling configured (min/max instances)
• SSL/TLS enabled on all endpoints

Monitoring

• Application metrics exported (request rate, latency, errors)
• Alerts configured for error rate > threshold
• Log aggregation set up (structured logs, searchable)
• Uptime monitoring on health endpoint

Security

• Dependencies scanned for CVEs
• CORS configured for allowed origins only
• Rate limiting enabled on public endpoints
• Authentication and authorization verified
• Security headers set (CSP, HSTS, X-Frame-Options)

Operations

• Rollback plan documented and tested
• Database migration tested against production-sized data
• Runbook for common failure scenarios
• On-call rotation and escalation path defined

---

GitHub Actions Standards

OIDC Authentication (No Static Credentials)

Use OIDC to authenticate to cloud providers, never store long-lived credentials in CI secrets:

jobs:
  deploy:
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/deploy-role
          aws-region: eu-west-1

Job-Level Permissions Block (Required)

Every job must declare minimum required permissions explicitly:

jobs:
  build:
    permissions:
      contents: read        # checkout
      packages: write       # push to GHCR
      id-token: write       # OIDC
      pull-requests: write  # PR comments

Never use permissions: write-all at workflow or job level.

Pipeline Linting

Add actionlint as a required CI check on all workflow changes:

- name: Lint GitHub Actions workflows
  uses: rhysd/actionlint@v1

Failure Notifications

On any pipeline failure, notify the PR author and a shared Slack channel:

- name: Notify on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    payload: |
      {
        "text": "Pipeline failed: ${{ github.workflow }} · ${{ github.job }}",
        "blocks": [{
          "type": "section",
          "text": {
            "type": "mrkdwn",
            "text": "*Pipeline:* ${{ github.workflow }}\n*Job:* ${{ github.job }}\n*Branch:* ${{ github.ref_name }}\n*SHA:* ${{ github.sha }}\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View logs>"
          }
        }]
      }

Matrix Build Configuration

strategy:
  fail-fast: true    # PR builds: stop on first failure
  # fail-fast: false  # Release validation: test all combinations
  matrix:
    os: [ubuntu-latest, windows-latest]
    java: ['21', '23']

Use fail-fast: true for PR builds; disable for release validation matrices.

Pipeline SLA

• CI (lint + test + build): <= 15 minutes
• Deployment pipeline (staging + production): <= 10 minutes
• If a pipeline exceeds SLA, raise it as a tech-debt issue within 2 business days

---

Canary Deployment

Traffic Rollout Schedule

Increment canary traffic in three steps with a health gate between each:

Step	Traffic	Minimum soak time
1	5%	10 minutes
2	25%	10 minutes
3	100%	,

Abort and roll back automatically if error rate exceeds baseline by > 1% or p99 latency increases > 20% during any soak period.

Automated Rollback

Automated rollback is a safety reflex and is acceptable even while production promotion stays a manual, approved gate: rolling back returns the system to the last known-good state, it does not push a new untested release forward.

Configure an automated rollback job triggered on health check failure or error rate spike detected within 60 seconds of traffic shift:

- name: Check canary health
  run: |
    ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_requests_total{status=~'5..'}[1m])" | jq '.data.result[0].value[1]' -r)
    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate $ERROR_RATE exceeds threshold — rolling back"
      kubectl rollout undo deployment/my-app
      exit 1
    fi

kubectl rollout undo is a last-resort manual fallback only. Rollbacks in production must be triggered by the automated pipeline, not manually by engineers.

---

Kubernetes Cluster Standards

Pod Security Context (Required on Every Deployment)

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: [ALL]

PodDisruptionBudget (Required for All Production Deployments)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 1      # or maxUnavailable: 1 — never allow all pods down simultaneously
  selector:
    matchLabels:
      app: my-app

NetworkPolicy, Default Deny

Every namespace must have a default-deny NetworkPolicy. Allowances are explicit:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-ingress
spec:
  podSelector:
    matchLabels:
      app: my-app
  ingress:
    - from:
        - podSelector:
            matchLabels:
              role: ingress-controller
      ports:
        - protocol: TCP
          port: 8080

Autoscaling

# HPA for request-driven scaling
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Use KEDA for event-driven workloads (queue depth, Kafka lag). Use VPA for resource right-sizing in non-production environments.

Resource Governance

Every namespace must have ResourceQuota and LimitRange:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - type: Container
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi

Observability

Every service must expose Prometheus metrics via a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      path: /actuator/prometheus
      interval: 15s

Policy Enforcement

Use Kyverno (preferred) or OPA Gatekeeper to enforce cluster policies:

• Require securityContext.runAsNonRoot: true
• Require resource requests/limits on all containers
• Block images without a digest or from non-approved registries
• Require PodDisruptionBudget for Deployments with replicas > 1

Stateful Workload Protection

# VolumeSnapshot before any stateful upgrade
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pre-upgrade-snapshot
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: my-data-pvc

Set reclaimPolicy: Retain on all production PersistentVolumes to prevent accidental data loss.

---

Helm Chart Standards

Required Files

Every Helm chart must contain:

• Chart.yaml: name, version, appVersion, description, maintainers
• values.yaml: all configurable values with inline comments
• templates/_helpers.tpl: shared named templates
• NOTES.txt: post-install instructions

Value Validation

Use required and fail functions to validate critical values at render time:

# templates/deployment.yaml
image:
  repository: {{ required "image.repository is required" .Values.image.repository }}
  tag: {{ required "image.tag is required" .Values.image.tag }}

Distribution & Deployment

• Distribute charts via an OCI registry (helm push, helm pull oci://)
• Commit Chart.lock to VCS
• Deploy with --atomic flag (auto-rollback on failure)
• Never store secrets as plaintext in chart YAML: use External Secrets Operator or Sealed Secrets
• Security scan charts with Trivy + Checkov before publishing

---

Cloud Provider Standards

IAM, Least Privilege

• No wildcard permissions (*) on any resource
• One service account per workload: never share between services
• Rotate service account keys every 90 days (or use Workload Identity / OIDC instead)

Network Isolation

• All databases and internal services in private subnets: no public endpoints
• Bastion/jump host or VPN required for admin access

Infrastructure as Code

• Remote Terraform state with locking: S3 + DynamoDB (AWS) or GCS (GCP)
• Never commit .tfstate files to VCS

Resource Tagging (Required)

Every cloud resource must have these tags:

Tag	Example
env	production
team	platform
service	auth-api
cost-center	engineering

Cost Controls

• Budget alerts at 80% and 100% of monthly budget
• Quota monitoring with alerts before hitting service limits

Audit & Security Posture

• CloudTrail (AWS) / Cloud Audit Logs (GCP) enabled in all accounts
• GuardDuty (AWS) / Security Command Center (GCP) enabled
• CDN configuration for all static assets

Disaster Recovery

• Multi-region DR with documented RTO and RPO targets
• Annual DR drill with documented results
• Automated failover for RTO < 1 hour; manual runbook for RTO >= 1 hour