Devops ci cd

DevOps & CI/CD

This skill assumes a Node.js backend. For Python/Go/Rust, adapt the specifics but the patterns transfer (test-before-deploy, multi-stage Docker, liveness vs readiness, automated rollback, secrets hygiene). Example commands use Fly.io as one concrete PaaS, but the same workflow applies to Railway, Render, Heroku, AWS, GCP, or any container platform.

Fail-Fast Rule

If any CI step fails, stop the pipeline and do not deploy. Never deploy untested code. Never skip health check validation. Never push secrets to git. A broken deploy to production with live traffic and stateful connections is catastrophic.

Cardinal Rule

Every push to main must pass tests and type-checking before reaching production. A deploy workflow that runs {{DEPLOY_COMMAND}} with zero validation is the single highest-risk gap in any infrastructure.

Audit Baseline: Current State

deploy.yml (minimal, no tests)

# ANTI-PATTERN -- {{PROJECT_PATH}}/.github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: superfly/flyctl-actions/setup-flyctl@master
      - run: flyctl deploy --remote-only
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Problems:

1. No npm ci -- dependencies not installed
2. No test run -- test suite never executes
3. No npx tsc --noEmit -- TypeScript errors not caught
4. No build validation before deploy
5. No rollback on failed health check
6. No deployment notification
7. Single step -- any failure is opaque

Dockerfile (single-stage, no cache optimization)

FROM node:20-slim
RUN apt-get update && apt-get install -y libjemalloc2
COPY package.json package-lock.json* ./
RUN npm install --omit=dev
COPY src ./src
COPY tsconfig.json ./
RUN npx esbuild "$ENTRY_POINT" --bundle --outdir=dist  # e.g. main.ts, app.ts
CMD ["node", "--enable-source-maps", "dist/index.js"]

Deploy config (basic, single health check)

• App: {{APP_NAME}}, region: primary
• VM: sized for workload
• Health check: GET /healthz every 15s, timeout 10s, grace 60s
• No readiness check, no liveness distinction

Target CI/CD Pipeline

GitHub Actions: Complete Pipeline

# .github/workflows/deploy.yml
name: Test & Deploy

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: TypeScript type check
        run: npx tsc --noEmit

      - name: Run tests
        run: npm test
        env:
          NODE_OPTIONS: '--max-old-space-size=4096'

      - name: Build validation
        run: npm run build

  deploy:
    needs: test
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    runs-on: ubuntu-latest
    timeout-minutes: 10
    environment: production
    steps:
      - uses: actions/checkout@v4

      # Example: Fly.io. Swap for Railway/Render/Heroku/AWS/GCP equivalents.
      - uses: superfly/flyctl-actions/setup-flyctl@master

      - name: Deploy
        run: ${{ env.DEPLOY_COMMAND }}   # e.g. flyctl deploy --remote-only
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
          DEPLOY_COMMAND: flyctl deploy --remote-only

      - name: Wait for health check
        run: |
          echo "Waiting 60s for startup grace period..."
          sleep 60
          for i in $(seq 1 10); do
            STATUS=$(curl -sf {{PRODUCTION_URL}}/healthz | jq -r '.status' 2>/dev/null || echo "unreachable")
            echo "Health check attempt $i: $STATUS"
            if [ "$STATUS" = "ok" ] || [ "$STATUS" = "healthy" ]; then
              echo "Health check passed"
              exit 0
            fi
            sleep 10
          done
          echo "Health check failed after 10 attempts"
          exit 1

      - name: Rollback on failure
        if: failure()
        run: |
          echo "Deploy failed -- rolling back to previous release"
          # Example (Fly.io): re-deploy the previous release image
          flyctl releases --json | jq -r '.[1].id' | xargs -I {} flyctl deploy --image registry.fly.io/{{APP_NAME}}:{}
          # Other platforms: use their equivalent rollback command
          # (Railway: railway rollback, Render: dashboard or API, Heroku: heroku releases:rollback, AWS: blue/green swap)
        env:
          FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}

Key Pipeline Principles

1. Test job is blocking: deploy has needs: test. No tests = no deploy.
2. PR validation: Runs tests on PRs without deploying (the if guard).
3. Concurrency control: Only one deploy at a time, newer cancels older.
4. Health verification: Post-deploy curl check with 10 retries.
5. Automatic rollback: If health check fails, revert to previous release.
6. Timeouts: 15 min for tests, 10 min for deploy.

Docker Multi-Stage Build Optimization

Current Problem

A naive Dockerfile installs ALL dependencies (npm install --omit=dev), copies source, and builds in a single stage. This means:

• Every source change invalidates the npm cache layer
• Dev dependencies needed for the build are installed then unused
• Final image includes npm, source files, and build artifacts

Optimized Multi-Stage Dockerfile

# Stage 1: Dependencies (cached unless package-lock changes)
FROM node:20-slim AS deps
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm ci

# Stage 2: Build (cached unless src changes)
FROM deps AS build
COPY src ./src
COPY tsconfig.json ./
RUN npm run build

# Stage 3: Production (minimal image)
FROM node:20-slim AS production
RUN apt-get update \
  && apt-get install -y --no-install-recommends libjemalloc2 curl \
  && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY package.json ./

ENV GIT_SHA=docker
ENV BUILD_TIME=unknown
EXPOSE 3000

HEALTHCHECK --interval=15s --timeout=10s --start-period=60s --retries=3 \
  CMD curl -f http://localhost:3000/healthz || exit 1

CMD ["node", "--enable-source-maps", "--max-old-space-size=8192", \
     "--max-semi-space-size=128", "dist/index.js"]

Benefits

• Layer caching: npm ci only re-runs when package-lock.json changes
• Smaller image: Production stage has no source files, no build tools
• Explicit healthcheck: Docker-level health monitoring
• curl included: For health check verification

Platform Configuration

Example: fly.toml (one PaaS among many)

The patterns below apply to any PaaS — just translate to the equivalent config format (Railway railway.toml, Render render.yaml, Heroku app.json, AWS ECS task def, GCP Cloud Run service YAML, etc.).

app = '{{APP_NAME}}'
primary_region = 'gru'

[env]
  PORT = "3000"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = 'off'
  auto_start_machines = true
  min_machines_running = 1
  processes = ['app']

[[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 2

# Liveness: is the process alive and responsive?
[checks.liveness]
  type = "http"
  port = 3000
  path = "/healthz"
  interval = "15s"
  timeout = "10s"
  grace_period = "120s"

# Readiness: is the system warmed up and serving accurate data?
[checks.readiness]
  type = "http"
  port = 3000
  path = "/readyz"
  interval = "30s"
  timeout = "10s"
  grace_period = "300s"

Health Check Design

/healthz -- Liveness (should the process be restarted?)

Returns 200 if the Node.js process is alive and the event loop is responsive. Returns 503 if the event loop is stalled or critical subsystems are down.

// CORRECT: Checks real health
app.get('/healthz', (c) => {
  const elStall = getEventLoopStallMs();
  const memUsage = process.memoryUsage();
  const heapPct = memUsage.heapUsed / memUsage.heapTotal;

  if (elStall > 5000) {
    return c.json({
      status: 'unhealthy',
      reason: 'event_loop_stall',
      el_stall_ms: elStall,
    }, 503);
  }

  if (heapPct > 0.95) {
    return c.json({
      status: 'unhealthy',
      reason: 'heap_exhaustion',
      heap_pct: heapPct,
    }, 503);
  }

  return c.json({ status: 'ok', uptime: process.uptime() });
});

// WRONG: Always returns 200
app.get('/healthz', (c) => c.json({ status: 'ok' }));

/readyz -- Readiness (should traffic be routed here?)

Returns 200 only when the system is warmed up with sufficient data/state. Returns 503 during the WARMING phase immediately after deploy.

app.get('/readyz', (c) => {
  const warmState = engine.getWarmStateMetrics();
  const readyPct = warmState.loaded / warmState.expected;

  if (readyPct < 0.5) {
    return c.json({
      status: 'warming',
      ready_pct: readyPct,
      loaded: warmState.loaded,
      expected: warmState.expected,
    }, 503);
  }

  return c.json({
    status: 'ready',
    loaded: warmState.loaded,
  });
});

Key distinction: A degraded system (some upstream dependencies down) is LIVE but should log warnings. A system that has not finished warming up is NOT READY and should not serve traffic.

Project-specific status/metrics endpoints (beyond /healthz and /readyz) can expose detailed internal state for operators, but keep them behind admin auth.

Rollback Strategies

Strategy 1: Platform Release Rollback (Fastest)

# Example: Fly.io
fly releases -a {{APP_NAME}}
fly releases rollback -a {{APP_NAME}}
fly deploy --image registry.fly.io/{{APP_NAME}}:<sha>

# Other platforms:
# Railway: railway rollback
# Render:  use dashboard or API to redeploy previous image
# Heroku:  heroku releases:rollback v<N> -a {{APP_NAME}}
# AWS ECS: update service to previous task definition
# GCP Cloud Run: gcloud run services update-traffic --to-revisions=<prev>=100

When to use: Bad deploy, immediate rollback needed. Risk: Low. Reverts to known-good image.

Strategy 2: Canary Deploy (Safest for Risky Changes)

# Example on Fly.io (translate to your PaaS):
fly scale count 2 -a {{APP_NAME}}
fly deploy --strategy rolling -a {{APP_NAME}}

# Monitor health of new machine
fly status -a {{APP_NAME}}
curl {{PRODUCTION_URL}}/healthz

# If healthy: scale back down
# If unhealthy: rollback immediately

When to use: Architecture changes, IPC changes, worker thread changes. Risk: Requires careful monitoring during rollout.

Strategy 3: Manual Revert (Git-Based)

# Revert the problematic commit
cd {{PROJECT_PATH}}
git revert HEAD
git push origin main
# CI/CD pipeline redeploys automatically

When to use: Code bug identified, clean revert possible. Risk: Revert may not be clean if commit has dependencies.

Stateful-System Rollback Concerns

• Stateful WebSocket / long-lived connections: All client connections reconnect on deploy. A 5-10 minute warm-up can be normal depending on workload. Do not rollback just because state counters are at 0 immediately after deploy.
• Worker threads / child processes: All worker threads and child processes restart. Expect 30-60s to stabilize.
• SharedArrayBuffer / in-memory caches: Reset on restart. Consumers must handle stale/zero values during warm-up.
• Grace period: The PaaS grace_period value MUST be longer than the expected warm-up time for basic health, or the platform will kill the machine before it reports healthy.

Secret Management

Runtime Secrets (PaaS-managed)

# Example: Fly.io (translate to your PaaS — Railway variables,
# Render env groups, Heroku config vars, AWS Secrets Manager / SSM,
# GCP Secret Manager, etc.)
fly secrets set DB_PASSWORD=<value> -a {{APP_NAME}}
fly secrets set AUTH_SECRET=<value> -a {{APP_NAME}}
fly secrets set THIRD_PARTY_API_KEY=<value> -a {{APP_NAME}}
# ...

# List secrets (names only, never values)
fly secrets list -a {{APP_NAME}}

GitHub Secrets (CI Pipeline)

Secret	Purpose	Used In
FLY_API_TOKEN (or equivalent)	Deploy authentication	deploy.yml

Rule: ONLY the deploy-auth token should be in GitHub Secrets. All other secrets live in the PaaS and are injected at runtime. Never duplicate database credentials, third-party keys, or provider API keys in GitHub.

Secrets Checklist

• No secrets in PaaS config (fly.toml [env], render.yaml envVars, etc.) — only non-sensitive config belongs there
• No secrets in .env committed to git (.gitignore must include .env*)
• No secrets in Dockerfile ENV or ARG
• No secrets in GitHub Actions workflow files
• Third-party API keys stored encrypted at rest in your DB if user-scoped
• Key rotation procedure documented

Monitoring Infrastructure

Prometheus + Grafana

Expose metrics from the backend for scraping:

// Expose /metrics endpoint for Prometheus
app.get('/metrics', adminAuthMiddleware, async (c) => {
  const metrics = collectMetrics(); // existing internal metrics
  return c.text(formatPrometheusExposition(metrics), 200, {
    'Content-Type': 'text/plain; version=0.0.4',
  });
});

Example Key Metrics

Metric	Type	Alert Threshold
el_stall_p99_ms	Gauge	> 500ms = WARN, > 2000ms = CRITICAL
heap_used_bytes	Gauge	> 80% of max = WARN
rss_bytes	Gauge	> 85% of VM memory = CRITICAL
ipc_messages_per_sec	Counter	backpressure risk threshold = WARN
ws_connections_active	Gauge	too low = CRITICAL (adapters down)
worker_queue_depth	Gauge	too high = WARN (processing lag)
http_request_duration_ms	Histogram	p99 > 1000ms = WARN
upstream_reconnects	Counter	> N/min per upstream = WARN

Alert Rules (PagerDuty/Slack)

# Critical: Page immediately
- alert: CoreWorkloadZero
  expr: core_workload_counter == 0
  for: 5m
  annotations:
    summary: "Core workload counter at zero for 5 minutes"
    action: "Check worker/adapter processes, verify upstream connections"

- alert: EventLoopStall
  expr: el_stall_p99_ms > 2000
  for: 2m
  annotations:
    summary: "Event loop stalled >2s"
    action: "Check IPC message volume, heap usage, GC pauses"

- alert: HeapExhaustion
  expr: heap_used_bytes / heap_max_bytes > 0.95
  for: 1m
  annotations:
    summary: "Heap at 95%, OOM imminent"
    action: "Check for memory leaks, restart if necessary"

# Warning: Slack notification
- alert: UpstreamDisconnected
  expr: upstream_connected == 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Upstream {{ $labels.upstream }} disconnected"
    action: "Check adapter logs, may self-recover via reconnect"

- alert: HighWorkerLag
  expr: worker_queue_depth > 500
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Worker queue depth >500, processing lagging"
    action: "Check throughput thresholds, consider scaling workers"

Deploy Checklist (Automated via CI)

Before every deploy, the pipeline must verify:

[ ] npm ci -- dependencies installed cleanly
[ ] npx tsc --noEmit -- zero TypeScript errors
[ ] npm test -- full test suite passes
[ ] Build step -- all entrypoints compile without errors
[ ] No secrets in committed files -- git-secrets scan
[ ] Health check responds 200 post-deploy
[ ] Readiness reports warm within expected window

Manual Deploy Checklist

When deploying manually (bypassing CI):

cd {{PROJECT_PATH}}

# 1. Run tests locally
npm test

# 2. Type check
npx tsc --noEmit

# 3. Commit and push
git add <specific files>
git commit -m "<message>"
git push origin main

# 4. Deploy (example on Fly.io; use your platform's equivalent)
{{DEPLOY_COMMAND}}   # e.g. fly deploy

# 5. Verify (wait 60s for warm-up)
sleep 60
curl {{PRODUCTION_URL}}/healthz
curl {{PRODUCTION_URL}}/readyz

# 6. Check status/metrics endpoint (behind admin auth)
curl -H "Authorization: Bearer <token>" \
  {{PRODUCTION_URL}}/admin/runtime

Zero-Downtime Deploy with WebSocket Connections

Challenge

A backend that maintains long-lived connections (WebSockets, SSE, gRPC streams) loses all of them on deploy unless handled carefully.

Mitigation Strategy

1. Rolling deploy: With 2+ machines, the PaaS drains connections from the old machine before stopping it. Clients reconnect to new machine. (Fly.io: --strategy rolling. Kubernetes: rolling update strategy. AWS ECS: minimum healthy percent + maximum percent.)

2. Client-side reconnect: Frontend/client must implement automatic reconnect with exponential backoff.

3. Grace period: A sufficient grace_period (e.g. 120s) gives the old machine time to drain existing connections before being killed.

4. SLO during deploy: Accept a short window of degraded data (WARMING status) as normal. The /readyz endpoint correctly reports this.

5. Upstream reconnection: Adapter processes restart all upstream connections. Expected warm-up depends on upstream count and priority ordering.

What NOT to Do

• Do not try to "hot swap" worker threads. Kill and restart is the correct approach for Node.js workers.
• Do not maintain two adapter processes simultaneously when upstream connections are unique per IP — two processes would cause conflicts.
• Do not rush the warm-up. Allow enough grace for the slowest upstream to come online before declaring the deploy bad.

Anti-Patterns to Reject

Anti-Pattern	Why It's Wrong	Correct Approach
Deploy without running tests	Broken code reaches production	needs: test in deploy job
/healthz always returns 200	Hides real failures, PaaS cannot auto-restart	Return 503 when degraded
Secrets in PaaS config or git	Exposed in repo history forever	PaaS secrets + GitHub secrets only
Single-stage Dockerfile	Slow builds, large images, poor caching	Multi-stage: deps, build, production
No rollback plan	Stuck with broken deploy	Automated rollback on health check failure
Deploying from local machine	Bypasses all CI checks	Always deploy via CI pipeline
Monitoring added after incident	Reactive, not proactive	Instrument from day one
Same health check for liveness and readiness	Different failure modes need different checks	/healthz (alive) vs /readyz (ready)
Ignoring warm-up period	False alerts on every deploy	Grace period in health checks + readiness check
Force-killing during deploy	Drops active connections ungracefully	Drain period + client reconnect logic

Hard-won lessons

• Dockerfile build entrypoint list drifts from package.json build script. When adding new worker files, ALWAYS update BOTH. A worker will crash with code:1 if the .js output is missing from dist/.
• fly logs --no-tail (and equivalent non-streaming log commands) return only a small tail buffer — use streaming logs or narrow with grep for debugging.
• TDZ in const declarations: JavaScript hoists const/let but they are NOT initialized until the declaration line. tsx/esbuild does NOT catch these at build time — they explode at runtime.
• V8 --max-old-space-size tuning matters: Undersized heaps can dramatically regress p50 latency due to GC pressure. Profile under realistic load and pick a size that gives headroom without oversubscribing the VM.