Sre canary

SRE Canary — Post-Deploy Monitoring & Canary Verification

Approach every post-deploy window as the engineer who will be paged if something breaks. A deployment is not done when code ships — it is done when you have confirmed that real production traffic is behaving the same as or better than before the change. Every minute of a degraded canary that you do not catch is a minute of user-visible failure that compounds.

The two failure modes are symmetric and equally dangerous: rolling back too early wastes a deploy and erodes team confidence; rolling forward on a broken canary causes real outages. This skill teaches you to read the signals precisely enough to tell them apart.

---

Step 0: Before You Start Monitoring

Before watching any signals, establish your baseline. Monitoring without a baseline is just staring at numbers.

1. Record the pre-deploy steady state — p50/p95/p99 latency, error rate, request rate, and Core Web Vitals from the last 30 minutes before the deploy.
2. Note the deploy timestamp exactly — you will overlay this on every graph. The most common source of confusion in post-deploy review is uncertainty about whether a signal shift happened before or after the change.
3. Identify the blast radius — which endpoints, services, user segments, or geographic regions does this deploy touch? Monitor those first; do not drown in unrelated signal.
4. Know your rollback command before you deploy — not after. The worst time to find your rollback runbook is during an active incident.
5. Define success criteria in advance — "error rate stays below 0.5%, p99 latency stays below 800ms, no new JS exceptions" is a success criterion. "Seems fine" is not.

---

Post-Deploy Monitoring Loop

Run this loop continuously during the canary window. Do not walk away after pressing deploy.

Phase 1: Immediate (0–5 minutes post-deploy)

The first five minutes catch hard failures: startup crashes, broken health checks, misconfigured routing, missing environment variables, and database migration errors.

What to check:

# 1. Confirm new pods/instances are running and healthy
kubectl get pods -n production -l app=<service> --watch

# 2. Check recent deployment events for errors
kubectl describe deployment <service> -n production | tail -30

# 3. Tail application logs for the new pods only
kubectl logs -n production -l app=<service>,version=<new-version> --since=5m -f

# 4. Watch the error rate in real time (adapt to your stack)
watch -n 5 'curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~\"5..\",service=\"<service>\"}[1m])" | jq .data.result[0].value[1]'

# 5. Confirm health check is passing on new instances
kubectl exec -n production deploy/<service> -- curl -sf http://localhost:8080/health

Rollback immediately if:

• Any new pod enters CrashLoopBackOff or Error state
• Health check returns non-200 on new instances
• Error rate jumps above 5× baseline within 2 minutes
• Application logs show unhandled exceptions on startup path

Phase 2: Stabilization (5–15 minutes post-deploy)

The process is running but we are watching for regressions that only appear under real traffic: slow queries exposed by a schema change, memory growth from a leak, latency spikes from a missing cache warm-up.

What to check:

# HTTP error rate by status code — differentiate 4xx (client) from 5xx (server)
rate(http_requests_total{service="<service>",status=~"5.."}[2m])
rate(http_requests_total{service="<service>",status=~"4.."}[2m])

# Latency percentiles — watch p99 specifically; it moves first
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="<service>"}[2m]))
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{service="<service>"}[2m]))
histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{service="<service>"}[2m]))

# Memory usage — a leak will show as monotonic growth here
container_memory_working_set_bytes{container="<service>", namespace="production"}

# CPU — sudden sustained spike on new version is a regression signal
rate(container_cpu_usage_seconds_total{container="<service>", namespace="production"}[2m])

# DB slow queries — a schema change or missing index will surface here
rate(db_query_duration_seconds_bucket{le="1",service="<service>"}[2m])

Alert thresholds — stabilization phase:

Signal	Warning	Critical
5xx error rate	> 2× baseline	> 5× baseline or > 1% absolute
p99 latency	> 1.5× baseline	> 2× baseline or > 2s absolute
p95 latency	> 1.3× baseline	> 1.75× baseline
Memory growth	> 20% above baseline	> 50% above baseline or OOM risk
CPU	> 30% above baseline	> 60% above baseline, sustained
New JS exceptions	Any new error type	Error rate > 1/min

Phase 3: Confidence Window (15–60 minutes post-deploy)

Traffic has flowed through the new version. You are now watching for issues that only emerge over time: connection pool saturation, cache eviction patterns, slow memory leaks, and long-tail edge cases.

Extended checks:

# Connection pool saturation
db_connection_pool_used{service="<service>"} / db_connection_pool_max{service="<service>"}

# Cache hit rate — a regression here causes latency spikes at the DB
cache_hits_total{service="<service>"} / (cache_hits_total + cache_misses_total)

# Queue depth — if the service processes async work, watch for backup
rabbitmq_queue_messages{queue="<service>-queue"}
# or
aws_sqs_approximate_number_of_messages_visible{QueueName="<service>-queue"}

# Downstream service health — your change may affect dependencies
rate(http_requests_total{client="<service>",status=~"5.."}[5m])

# Apdex score — composite satisfaction metric
(
  rate(http_request_duration_seconds_bucket{le="0.3",service="<service>"}[5m]) +
  rate(http_request_duration_seconds_bucket{le="1.2",service="<service>"}[5m]) / 2
) / rate(http_request_duration_seconds_count{service="<service>"}[5m])

---

Canary Deployment Verification

A canary routes a small percentage of production traffic to the new version while the old version handles the rest. You compare the two cohorts directly — same traffic, same load, same conditions — to detect regressions with statistical confidence before full rollout.

Traffic Split Configuration

Kubernetes with Argo Rollouts:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-service
  namespace: production
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: my-service-canary
      stableService: my-service-stable
      trafficRouting:
        nginx:
          stableIngress: my-service-ingress
      steps:
        - setWeight: 5       # Step 1: 5% canary traffic
        - pause: {duration: 10m}
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 20      # Step 2: 20% canary traffic
        - pause: {duration: 15m}
        - analysis:
            templates:
              - templateName: success-rate
              - templateName: latency-p99
        - setWeight: 50      # Step 3: 50% canary traffic
        - pause: {duration: 20m}
        - analysis:
            templates:
              - templateName: success-rate
              - templateName: latency-p99
              - templateName: error-budget
        # Step 4: 100% — promoted if all analyses pass
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service

AnalysisTemplate — success rate:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: production
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.995
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="my-service-canary",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="my-service-canary"}[2m]))

AnalysisTemplate — p99 latency:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: latency-p99
  namespace: production
spec:
  metrics:
    - name: latency-p99
      interval: 1m
      successCondition: result[0] <= 0.8   # 800ms
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{service="my-service-canary"}[2m]))
              by (le)
            )

Canary Rollout Gates — 5% → 20% → 50% → 100%

Each gate is a set of conditions that must be true before advancing. Treat them as mandatory, not advisory.

Gate 1: 5% → 20% (after 10 minutes at 5%)

Pass conditions:

• 5xx error rate on canary ≤ baseline × 1.2 (20% tolerance)
• p99 latency on canary ≤ baseline × 1.2
• No new unhandled exception types in canary logs
• All canary instances passing health checks
• Memory usage on canary ≤ baseline × 1.3

Fail conditions (auto-rollback):

• 5xx rate on canary > 5× baseline for 2+ consecutive minutes
• p99 latency > 2× baseline for 3+ consecutive minutes
• Any canary pod in CrashLoopBackOff
• Health check failure rate > 0

Gate 2: 20% → 50% (after 15 minutes at 20%)

Pass conditions (all Gate 1 conditions, plus):

• Apdex score on canary ≥ Apdex score on stable × 0.98
• DB query p95 on canary ≤ baseline × 1.15
• Cache hit rate on canary ≥ baseline × 0.95
• No alerts firing on canary-specific dashboards
• Queue depth (if applicable) not growing on canary traffic

Fail conditions (auto-rollback):

• Apdex degradation > 5% from stable
• Any memory leak signal (sustained growth > 20% over 10 minutes)
• Error budget burn rate > 3× normal

Gate 3: 50% → 100% (after 20 minutes at 50%)

Pass conditions (all previous, plus):

• Error budget consumed during canary window < 10% of hourly budget
• No customer-escalated incidents attributable to canary
• SLO compliance rate ≥ SLO target
• All Core Web Vitals within acceptable range (if frontend change)
• No anomalous patterns in structured logs (no new error codes, unusual request patterns)
• Downstream service health unaffected by canary traffic

Promotion command:

# Argo Rollouts
kubectl argo rollouts promote my-service -n production

# Verify promotion
kubectl argo rollouts status my-service -n production

Rollback command:

# Argo Rollouts — abort and return all traffic to stable
kubectl argo rollouts abort my-service -n production
kubectl argo rollouts undo my-service -n production

# Verify rollback
kubectl argo rollouts get rollout my-service -n production --watch

---

Health Check Patterns

Health checks are the foundation of every automated canary gate. A misconfigured health check gives you false confidence during the canary window.

Liveness vs. Readiness vs. Startup

• Liveness: Is the process alive and not deadlocked? If this fails, the container is restarted. Liveness checks should be simple — if they are expensive, a degraded service will fail the check, restart, and oscillate.
• Readiness: Is the instance ready to receive traffic? If this fails, the instance is removed from the load balancer without being restarted. Use readiness to signal warm-up, dependency unavailability, or overload.
• Startup: Has the application finished initializing? Prevents premature liveness checks from killing slow-starting containers.

# Kubernetes probe configuration — production-grade
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3    # 30 seconds of failure before restart

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2    # 10 seconds before removed from LB

startupProbe:
  httpGet:
    path: /health/live
    port: 8080
  failureThreshold: 30
  periodSeconds: 5       # Allow up to 150 seconds for startup

Health Endpoint Implementation

# Python / FastAPI example — production health endpoint
from fastapi import FastAPI, status
from fastapi.responses import JSONResponse
import time

app = FastAPI()
START_TIME = time.time()

@app.get("/health/live")
async def liveness():
    """
    Liveness: is the process functional?
    Only check internal process state — not external dependencies.
    A liveness failure causes a restart; do not be trigger-happy.
    """
    return JSONResponse(
        status_code=status.HTTP_200_OK,
        content={"status": "live", "uptime_seconds": int(time.time() - START_TIME)}
    )

@app.get("/health/ready")
async def readiness():
    """
    Readiness: can this instance serve traffic?
    Check critical dependencies. A failure removes from LB without restart.
    """
    checks = {}
    overall_ok = True

    # Database connectivity
    try:
        await db.execute("SELECT 1")
        checks["database"] = "ok"
    except Exception as e:
        checks["database"] = f"error: {str(e)}"
        overall_ok = False

    # Cache connectivity
    try:
        await redis.ping()
        checks["cache"] = "ok"
    except Exception as e:
        checks["cache"] = f"error: {str(e)}"
        # Degrade gracefully — cache failure may not require removing from LB
        # Adjust based on your caching criticality

    http_status = status.HTTP_200_OK if overall_ok else status.HTTP_503_SERVICE_UNAVAILABLE
    return JSONResponse(status_code=http_status, content={"status": "ready" if overall_ok else "not_ready", "checks": checks})

Synthetic Health Checks (Active Probing)

In addition to Kubernetes probes, run active synthetic checks that simulate real user flows:

# Canary synthetic check script — run every 60 seconds during canary window
#!/bin/bash
set -e

CANARY_HOST="${CANARY_HOST:-https://canary.example.com}"
STABLE_HOST="${STABLE_HOST:-https://www.example.com}"
MAX_LATENCY_MS="${MAX_LATENCY_MS:-800}"
ERROR_THRESHOLD="${ERROR_THRESHOLD:-0.01}"

check_endpoint() {
  local host="$1"
  local path="$2"
  local label="$3"

  START=$(date +%s%3N)
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "${host}${path}")
  END=$(date +%s%3N)
  DURATION=$((END - START))

  echo "[${label}] ${path} → HTTP ${HTTP_CODE} in ${DURATION}ms"

  if [[ "$HTTP_CODE" -ge 500 ]]; then
    echo "FAIL: 5xx on ${label} ${path}"
    return 1
  fi

  if [[ "$DURATION" -gt "$MAX_LATENCY_MS" ]]; then
    echo "WARN: Latency ${DURATION}ms exceeds ${MAX_LATENCY_MS}ms threshold on ${label} ${path}"
  fi
}

# Run the same checks against canary and stable
for path in "/" "/api/products" "/api/users/me" "/api/search?q=test"; do
  check_endpoint "$CANARY_HOST" "$path" "CANARY"
  check_endpoint "$STABLE_HOST" "$path" "STABLE"
  echo "---"
done

---

Alert Thresholds and Rollback Triggers

Error Rate Thresholds

Define error budgets before the deploy, not during the incident.

SLO Target	Error Budget / 30 days	Max hourly burn	Auto-rollback threshold
99.9%	43.8 minutes	~1.5 minutes	5xx rate > 0.5% for 3 min
99.5%	3.65 hours	~7.5 minutes	5xx rate > 1.0% for 3 min
99.0%	7.3 hours	~15 minutes	5xx rate > 2.0% for 5 min

Canary-specific alert: error rate comparison

The most actionable canary alert is not the absolute error rate — it is the ratio of canary error rate to stable error rate. A 0.5% error rate is fine if the stable version also has 0.5%. A 0.5% error rate is a regression if the stable version has 0.05%.

# Alert when canary error rate is more than 3× stable error rate
(
  rate(http_requests_total{service="my-service-canary",status=~"5.."}[5m])
  /
  rate(http_requests_total{service="my-service-canary"}[5m])
)
/
(
  rate(http_requests_total{service="my-service-stable",status=~"5.."}[5m])
  /
  rate(http_requests_total{service="my-service-stable"}[5m])
) > 3

Latency Thresholds

# Alert when canary p99 latency is 50% worse than stable p99
(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="my-service-canary"}[5m]))
)
/
(
  histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="my-service-stable"}[5m]))
) > 1.5

Automatic Rollback Triggers

Define these before deploying. Automation is faster than humans when something breaks at 3am.

Hard rollback triggers (automated, immediate):

• 5xx error rate on canary > 5% absolute for 2 consecutive minutes
• p99 latency on canary > 3× baseline for 3 consecutive minutes
• Any canary pod in CrashLoopBackOff
• Health check failure rate on canary > 0 for 1 minute
• Error budget burn rate > 10× normal (consuming an hour's budget in 6 minutes)

Soft rollback triggers (alert + human decision within 5 minutes):

• 5xx error rate on canary > 2× stable for 5 minutes
• p99 latency on canary > 1.5× stable for 5 minutes
• New JavaScript exception type appearing at rate > 1/minute
• Cache hit rate drops > 10% on canary
• Apdex degradation > 3% vs. stable

# Manual rollback — always test this before you need it
kubectl argo rollouts abort my-service -n production && \
kubectl argo rollouts undo my-service -n production

# Verify traffic is back on stable
kubectl argo rollouts get rollout my-service -n production
kubectl get pods -n production -l app=my-service

# Confirm error rate is recovering
watch -n 10 'kubectl top pods -n production -l app=my-service'

---

SLO/SLA Monitoring During Canary

Error Budget Burn Rate

Burn rate is the rate at which you are consuming your monthly error budget. A burn rate of 1 means you are burning exactly at budget — you will use up the month's budget by end of month. A burn rate of 10 means you will exhaust the budget in 1/10 of the month (about 3 days).

During a canary, burn rate spikes are your highest-signal alert.

# 5-minute burn rate — short window, fast response
(
  1 - (
    rate(http_requests_total{service="my-service",status!~"5.."}[5m])
    /
    rate(http_requests_total{service="my-service"}[5m])
  )
) / (1 - 0.999)  # Replace 0.999 with your SLO target

# 1-hour burn rate — medium window, catches gradual degradation
(
  1 - (
    rate(http_requests_total{service="my-service",status!~"5.."}[1h])
    /
    rate(http_requests_total{service="my-service"}[1h])
  )
) / (1 - 0.999)

Burn rate alert thresholds (Google SRE Workbook recommendations):

Window	Burn Rate	Action
5 minutes	> 14.4×	Page immediately — critical
1 hour	> 14.4×	Page immediately — critical
6 hours	> 6×	Ticket + investigate
3 days	> 3×	Review and plan

SLO Dashboard During Canary Window

A canary-window SLO dashboard should show, side-by-side:

1. Error rate: canary vs. stable — the comparison is more informative than the absolute value
2. Latency percentiles: canary vs. stable — p50, p95, p99, all on the same chart
3. Error budget remaining — how much of the monthly budget is left
4. Current burn rate — 5-minute and 1-hour windows
5. Apdex score: canary vs. stable
6. Request rate: canary vs. stable — to verify traffic is actually being split as configured

If your observability platform is Grafana, add a deploy marker annotation:

# Post a Grafana deploy annotation — creates a vertical line on all dashboards
curl -s -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${GRAFANA_API_KEY}" \
  "${GRAFANA_URL}/api/annotations" \
  -d "{
    \"dashboardId\": ${DASHBOARD_ID},
    \"time\": $(date +%s000),
    \"text\": \"Deploy: ${SERVICE} ${VERSION} — Canary ${CANARY_WEIGHT}%\",
    \"tags\": [\"deploy\", \"canary\", \"${SERVICE}\"]
  }"

---

Core Web Vitals Monitoring

For frontend deployments, Core Web Vitals are the highest-signal user-experience metrics. A canary that degrades CWV is hurting real users even if the API error rate is zero.

Thresholds

Metric	Good	Needs Improvement	Poor (Rollback)
LCP (Largest Contentful Paint)	< 2.5s	2.5s – 4.0s	> 4.0s
INP (Interaction to Next Paint)	< 200ms	200ms – 500ms	> 500ms
CLS (Cumulative Layout Shift)	< 0.1	0.1 – 0.25	> 0.25
FCP (First Contentful Paint)	< 1.8s	1.8s – 3.0s	> 3.0s
TTFB (Time to First Byte)	< 800ms	800ms – 1800ms	> 1800ms

Real User Monitoring (RUM) During Canary

Collect Core Web Vitals from real users in the canary cohort using the web-vitals library:

// Install: npm install web-vitals
import { onCLS, onINP, onLCP, onFCP, onTTFB } from 'web-vitals';

function sendToAnalytics({ name, value, id, delta, rating }) {
  // Tag with canary version so you can segment in your analytics
  const payload = {
    metric: name,
    value: Math.round(name === 'CLS' ? value * 1000 : value),
    delta: Math.round(name === 'CLS' ? delta * 1000 : delta),
    id,
    rating,                             // 'good' | 'needs-improvement' | 'poor'
    app_version: window.__APP_VERSION__, // injected at build time
    canary: window.__IS_CANARY__,       // true for canary instances
    page: window.location.pathname,
  };

  // Send to your analytics endpoint
  if (navigator.sendBeacon) {
    navigator.sendBeacon('/api/vitals', JSON.stringify(payload));
  } else {
    fetch('/api/vitals', { method: 'POST', body: JSON.stringify(payload), keepalive: true });
  }
}

onCLS(sendToAnalytics);
onINP(sendToAnalytics);
onLCP(sendToAnalytics);
onFCP(sendToAnalytics);
onTTFB(sendToAnalytics);

Segment your CWV data by canary flag in your analytics platform. The rollback signal is not just "LCP got worse" — it is "LCP is worse for the canary cohort and stable for the stable cohort."

Lighthouse CI in the Canary Pipeline

# .github/workflows/canary-cwv-check.yml
name: Core Web Vitals — Canary Check

on:
  workflow_dispatch:
    inputs:
      canary_url:
        description: 'Canary URL to check'
        required: true
      stable_url:
        description: 'Stable URL to compare against'
        required: true

jobs:
  cwv-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Lighthouse against canary
        uses: treosh/lighthouse-ci-action@v11
        with:
          urls: ${{ github.event.inputs.canary_url }}
          configPath: '.lighthouserc.json'
          uploadArtifacts: true
          temporaryPublicStorage: true

      - name: Compare results
        run: |
          # Extract LCP, CLS, INP from LHCI output and compare to baseline
          node scripts/compare-cwv.js --canary=lhci-results.json --threshold=10

---

Log Analysis During Canary Window

What to Watch in Logs

Structure your logs as JSON from day one. Canary log analysis on unstructured text is slow and error-prone.

Critical log signals during canary:

# 1. New error types not seen in stable — the most important signal
# Compare error.code or error.type distribution between canary and stable
kubectl logs -n production -l version=canary --since=15m \
  | jq -r 'select(.level=="error") | .error.code // .error.type // "unknown"' \
  | sort | uniq -c | sort -rn

# 2. Exception stack traces — new ones indicate new code paths failing
kubectl logs -n production -l version=canary --since=15m \
  | jq -r 'select(.level=="error") | .error.stack // empty' \
  | head -50

# 3. Slow operations — queries, external calls, serialization
kubectl logs -n production -l version=canary --since=15m \
  | jq 'select(.duration_ms > 500) | {path: .path, duration_ms, trace_id}'

# 4. Warning-level signals that precede errors
kubectl logs -n production -l version=canary --since=15m \
  | jq -r 'select(.level=="warn") | .message' \
  | sort | uniq -c | sort -rn

# 5. Correlation IDs for end-to-end trace reconstruction
# If you see an error in logs, pull the full trace:
kubectl logs -n production -l app=my-service --since=1h \
  | jq --arg tid "your-trace-id" 'select(.trace_id==$tid)'

Log Anomaly Detection

During the canary window, compare log volume and error distribution between canary and stable:

#!/bin/bash
# canary-log-compare.sh — run during canary window
NAMESPACE="${1:-production}"
SERVICE="${2:-my-service}"
WINDOW="${3:-15m}"

echo "=== Error distribution: CANARY ==="
kubectl logs -n "$NAMESPACE" -l "app=${SERVICE},version=canary" --since="$WINDOW" \
  | jq -r 'select(.level=="error") | .error_code // .message // "unclassified"' \
  | sort | uniq -c | sort -rn | head -20

echo ""
echo "=== Error distribution: STABLE ==="
kubectl logs -n "$NAMESPACE" -l "app=${SERVICE},version=stable" --since="$WINDOW" \
  | jq -r 'select(.level=="error") | .error_code // .message // "unclassified"' \
  | sort | uniq -c | sort -rn | head -20

echo ""
echo "=== Slow requests (>500ms): CANARY ==="
kubectl logs -n "$NAMESPACE" -l "app=${SERVICE},version=canary" --since="$WINDOW" \
  | jq -c 'select(.duration_ms != null and .duration_ms > 500) | {path, duration_ms, method, status}' \
  | head -20

echo ""
echo "=== NEW error types in CANARY not seen in STABLE ==="
CANARY_ERRORS=$(kubectl logs -n "$NAMESPACE" -l "app=${SERVICE},version=canary" --since="$WINDOW" \
  | jq -r 'select(.level=="error") | .error_code // empty' | sort -u)
STABLE_ERRORS=$(kubectl logs -n "$NAMESPACE" -l "app=${SERVICE},version=stable" --since="$WINDOW" \
  | jq -r 'select(.level=="error") | .error_code // empty' | sort -u)
comm -23 <(echo "$CANARY_ERRORS") <(echo "$STABLE_ERRORS")

Structured Logging Best Practices During Canary

Tag every log line with the version/canary identifier:

import logging
import json
import os
import time

class StructuredLogger:
    def __init__(self, service_name: str):
        self.service = service_name
        self.version = os.getenv("APP_VERSION", "unknown")
        self.is_canary = os.getenv("IS_CANARY", "false").lower() == "true"

    def _log(self, level: str, message: str, **kwargs):
        entry = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            "level": level,
            "message": message,
            "service": self.service,
            "version": self.version,
            "canary": self.is_canary,
            **kwargs
        }
        print(json.dumps(entry), flush=True)

    def info(self, message: str, **kwargs):
        self._log("info", message, **kwargs)

    def error(self, message: str, error: Exception = None, **kwargs):
        error_info = {}
        if error:
            error_info = {
                "error": {
                    "type": type(error).__name__,
                    "message": str(error),
                    "code": getattr(error, "code", None),
                }
            }
        self._log("error", message, **{**error_info, **kwargs})

---

Console Error Monitoring

For frontend deployments, JavaScript console errors are a leading indicator of user-facing failures. Monitor them during the canary window.

Browser Error Capture

// error-capture.js — load early in your bundle, before other scripts
(function () {
  const APP_VERSION = window.__APP_VERSION__ || 'unknown';
  const IS_CANARY = window.__IS_CANARY__ || false;

  // Capture unhandled JavaScript errors
  window.addEventListener('error', function (event) {
    reportError({
      type: 'js_error',
      message: event.message,
      source: event.filename,
      line: event.lineno,
      column: event.colno,
      stack: event.error?.stack,
    });
  });

  // Capture unhandled promise rejections
  window.addEventListener('unhandledrejection', function (event) {
    reportError({
      type: 'unhandled_rejection',
      message: String(event.reason),
      stack: event.reason?.stack,
    });
  });

  // Capture failed resource loads (images, scripts, stylesheets)
  window.addEventListener('error', function (event) {
    if (event.target && event.target.tagName) {
      reportError({
        type: 'resource_error',
        resource_type: event.target.tagName.toLowerCase(),
        resource_src: event.target.src || event.target.href,
      });
    }
  }, true /* capture phase to catch resource errors */);

  function reportError(data) {
    const payload = {
      ...data,
      app_version: APP_VERSION,
      canary: IS_CANARY,
      page: window.location.pathname,
      user_agent: navigator.userAgent,
      timestamp: new Date().toISOString(),
    };

    // Non-blocking — use sendBeacon to not impact page performance
    navigator.sendBeacon('/api/errors/browser', JSON.stringify(payload));
  }
})();

API Error Rate Monitoring

Track client-side API call failures — these are distinct from server-side 5xx rates because they include network errors and client-side request cancellations:

// api-monitor.js — wrap your fetch client
const originalFetch = window.fetch;
window.fetch = async function (url, options = {}) {
  const start = performance.now();

  try {
    const response = await originalFetch(url, options);
    const duration = performance.now() - start;

    if (!response.ok && response.status >= 500) {
      reportApiError({
        url: typeof url === 'string' ? url : url.url,
        method: options.method || 'GET',
        status: response.status,
        duration_ms: Math.round(duration),
        type: 'server_error',
      });
    }

    if (duration > 3000) {
      reportSlowRequest({
        url: typeof url === 'string' ? url : url.url,
        duration_ms: Math.round(duration),
      });
    }

    return response;
  } catch (error) {
    reportApiError({
      url: typeof url === 'string' ? url : url.url,
      method: options.method || 'GET',
      error: error.message,
      type: 'network_error',
    });
    throw error;
  }
};

---

Canary Runbook

Keep this runbook open and ready during every canary deploy.

Before Deploy

• Baseline metrics recorded (error rate, p99 latency, Apdex, CWV if frontend)
• Rollback command tested in staging within the last 30 days
• On-call engineer notified that a canary deploy is in progress
• Deploy annotation sent to monitoring platform
• Canary success criteria documented and agreed upon

During Deploy — 5% Phase (0–10 min)

• New pods healthy (kubectl get pods)
• Error rate on canary ≤ 1.2× baseline
• p99 latency on canary ≤ 1.2× baseline
• No new exception types in canary logs
• Health checks passing on all canary instances

During Deploy — 20% Phase (10–25 min)

• All 5% gate checks still passing
• Apdex score on canary ≥ 98% of stable
• DB query latency within 15% of baseline
• Cache hit rate within 5% of baseline
• No customer support escalations

During Deploy — 50% Phase (25–45 min)

• All previous gate checks still passing
• Error budget burn rate < 3× normal
• CWV within threshold (frontend only)
• Queue depths stable (async workers)
• No anomalies in log comparison (canary vs. stable)

Promote to 100%

• All gate checks passed at 50% phase
• SLO compliance rate ≥ SLO target throughout canary window
• No open incidents attributable to canary
• Post-deploy annotation updated in monitoring platform

Rollback Decision Tree

Error rate spike detected
│
├─ Is it isolated to canary cohort? ──── NO ──→ Pre-existing issue, not canary. Investigate separately.
│
└─ YES
   │
   ├─ How severe?
   │   ├─ > 5× baseline → ROLLBACK IMMEDIATELY (automated or manual)
   │   ├─ 2–5× baseline → Alert team, watch for 3 minutes, rollback if not improving
   │   └─ 1.2–2× baseline → Investigate root cause, continue at current weight, do not advance
   │
   └─ After rollback:
       ├─ Confirm error rate returning to stable baseline within 3 minutes
       ├─ Write incident timeline with deploy timestamp and rollback timestamp
       └─ Do not re-deploy without root cause identified

---

Definition of Done — Canary Verification

A canary is complete and the deploy is done when ALL of these are true:

• Traffic at 100% — no canary split remaining
• Error rate at 100% traffic is ≤ pre-deploy baseline (not just ≤ gate threshold)
• p99 latency at 100% traffic is ≤ pre-deploy baseline
• Error budget consumed during canary window < 5% of daily budget
• Zero new unhandled exception types introduced
• Core Web Vitals stable or improved (frontend deployments)
• Log anomaly check clean — no new error codes vs. stable
• All synthetic health checks passing
• Deploy annotation in monitoring platform with final status (promoted / rolled back)
• Rollback runbook confirmed to be current and tested
• Post-deploy monitoring continued for 30 minutes at 100% before declaring done
• If any gate required a hold: root cause documented even if deploy ultimately succeeded