Deployment strategies

是什么

这是一份发版策略规范，覆盖蓝绿、金丝雀、滚动、特性开关等模式的选择标准、回滚条件、数据库迁移协调流程，让团队既能高频发版又能在出问题时分钟级回滚。

怎么用

1. 新功能上线前，按本文档的风险矩阵决定走蓝绿、金丝雀还是滚动发布。
2. 大流量功能用 feature flag（特性开关）把部署和放量解耦，先 ship 代码再控开关。
3. 涉及数据库 schema（表结构）变更时，严格按文档的双写、读迁移、清理三阶段流程推进。
4. 每次发版前定义清楚回滚指标和阈值，触发即回滚，避免靠人脑临场拍板。
5. 发版后 30 分钟内盯紧错误率、延迟、流量三大指标，超出阈值立刻按预案处置。

架构图

flowchart LR
    A[新版本就绪] --> B[选定策略]
    B --> C[小流量灰度]
    C --> D{指标达标?}
    D -->|是| E[全量放开]
    D -->|否| F[自动回滚]

Deployment Strategies

A reference for selecting and implementing deployment strategies that minimize risk, enable zero-downtime releases, and provide fast rollback paths.

When to Activate

• Planning a deployment strategy for a new service or major release
• Implementing feature flags in an application
• Coordinating a database migration with a zero-downtime deployment
• Setting up canary releases or progressive delivery
• Defining rollback procedures for a service
• Reducing deployment risk for a high-traffic service

Strategy Comparison

Strategy	Traffic routing	Rollback speed	Risk	Infrastructure cost	Best for
Recreate	Stop all, start new	Fast (redeploy)	High (downtime)	Low	Dev/non-prod
Rolling update	Replace pods gradually	Medium (rollback flag)	Medium	Low	Most services
Blue/Green	Flip all traffic at once	Instant (flip back)	Low	2x	High-stakes releases
Canary	Shift % traffic gradually	Instant (shift back)	Very low	Slightly > 1x	High-traffic, data-sensitive
A/B Testing	Route by user segment	Instant	Low	~1x	Feature experiments
Shadow	Mirror traffic, no user impact	N/A	None	~2x	Testing new version with real traffic

Rolling Updates (Kubernetes)

Default Kubernetes behavior when you run kubectl apply. Pods are replaced incrementally — no full restart required.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # max pods above desired count during rollout
    maxUnavailable: 0  # never go below desired count (zero-downtime)

• Set maxUnavailable: 0 to guarantee zero downtime — new pods must pass readiness probes before old pods are terminated.
• Rollback: kubectl rollout undo deployment/my-service
• Target a specific revision: kubectl rollout undo deployment/my-service --to-revision=3
• Monitor progress: kubectl rollout status deployment/my-service
• Issue: slow rollback if many replicas; new version runs alongside old — both app versions must be compatible with current DB schema.

Blue/Green Deployments

Two identical environments run in parallel: Blue (live) and Green (new version). Traffic flips atomically from one to the other.

Process

1. Deploy new version to Green environment
2. Run smoke tests against Green (no user traffic yet)
3. Flip traffic: update load balancer rule or Kubernetes Service selector
4. Monitor error rate and latency for 15–30 minutes
5. Decommission Blue (or keep as instant rollback for 24 hours)

Kubernetes Implementation

Flip the Service selector to switch which deployment receives traffic.

# Blue deployment (live)
spec:
  selector:
    app: payment-service
    version: blue   # Service points here

# Green deployment (new)
spec:
  selector:
    app: payment-service
    version: green  # Update Service to point here after smoke tests

Flip command:

kubectl patch service payment-service -p '{"spec":{"selector":{"version":"green"}}}'

Considerations

• Cost: 2x infrastructure during transition window.
• Warm-up: Green must receive warming traffic (health checks, cache pre-warming) before the flip to avoid cold-start latency spikes.
• Database: Both Blue and Green versions must be compatible with the same DB schema during the transition window. Use the expand-contract pattern for migrations.

Canary Releases

Gradually shift traffic from the stable version to the new version. Automated analysis gates promotion based on SLO metrics.

• Typical progression: 5% → 25% → 50% → 100%
• Automated promotion: if error rate < 1% and p99 latency < 500 ms, advance
• Manual gate: require human approval before advancing beyond 25%
• Automated abort: if metrics breach thresholds, roll back instantly

Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: {}  # manual gate — requires human approval
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
      analysis:
        templates:
          - templateName: error-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: payment-service

Promote or abort the rollout:

kubectl argo rollouts promote payment-service  # advance to next step
kubectl argo rollouts abort payment-service    # rollback to stable

Flagger (Linkerd / Istio)

Flagger integrates with service meshes for automatic traffic splitting and metric-based promotion. Define a Canary CR with analysis.metrics referencing Prometheus queries. Flagger handles weight increments and rollback automatically — no manual step definitions required.

Feature Flags

Why Feature Flags

• Decouple deployment from release: deploy code, enable for users later
• Progressive rollout: enable for 1% → 10% → 100% of users without redeploying
• Kill switch: disable instantly without a deployment or rollback
• A/B testing: different experiences for user segments based on targeting rules

Flag Lifecycle

1. Add flag (disabled by default)
2. Deploy code wrapped behind flag
3. Enable for internal users → beta users → percentage rollout → 100%
4. Remove flag and dead code (flags are technical debt — clean up within a sprint of full rollout)

Tools Comparison

Tool	Hosting	SDK support	Best for
LaunchDarkly	Cloud (paid)	20+ SDKs	Enterprise, A/B testing
Unleash	Self-hosted or cloud	10+ SDKs	Open-source, full control
OpenFeature	Standard (vendor-agnostic SDK)	All vendors	Portability across providers
AWS AppConfig	Cloud	AWS SDK	AWS-native workloads
Environment variables	N/A	Simple	Simple boolean flags, no runtime toggle needed

Code Pattern (OpenFeature)

import { OpenFeature } from '@openfeature/server-sdk';

const client = OpenFeature.getClient();

// Simple boolean flag
const isNewCheckoutEnabled = await client.getBooleanValue(
  'new-checkout-flow',
  false,  // default value — returned if flag is missing or evaluation fails
  { targetingKey: userId }
);

if (isNewCheckoutEnabled) {
  return newCheckoutHandler(req, res);
} else {
  return legacyCheckoutHandler(req, res);
}

OpenFeature's provider abstraction means swapping from LaunchDarkly to Unleash requires changing only the registered provider — application code stays the same.

Database Migrations and Zero-Downtime Deployments

The Problem

Direct ALTER TABLE can lock tables under load. Renaming columns breaks the old app version that runs alongside the new version during a rolling deploy. Any migration that removes or renames a column must be done in phases.

Expand-Contract Pattern (Parallel Change)

Use for: adding NOT NULL columns, renaming columns or tables, changing data types.

Phase 1 — Expand (additive only):

• Add new column as NULLABLE
• Deploy application code that writes to both old and new columns
• No downtime — old app version still works with the old column

Phase 2 — Migrate:

• Backfill existing rows in batches to avoid table locks:

  UPDATE table SET new_col = old_col WHERE new_col IS NULL LIMIT 10000;
  

• Deploy application code that reads from the new column
• Add NOT NULL constraint once all rows are populated (now safe)

Phase 3 — Contract (remove old):

• Deploy application code that no longer references the old column
• Drop old column in a separate migration
• Can be done in a later sprint once confidence is high

Example Timeline

Renaming user.username to user.display_name:

Sprint 1: Add display_name (nullable), write to both columns
Sprint 2: Backfill rows, read from display_name, add NOT NULL
Sprint 3: Remove username column

Large Table Migrations

For tables with millions of rows, use pt-online-schema-change (Percona) or gh-ost (GitHub) to perform the migration on a shadow table and cut over with minimal locking.

Rollback Procedures

When to Roll Back

Roll back when:

• Error rate exceeds SLO threshold (e.g., > 1% errors) within 15 minutes of deploy
• p99 latency increases more than 2x baseline
• Critical functionality is broken (payments, login, data integrity)

Do not roll back immediately for:

• Cosmetic issues or minor UI regressions
• Minor performance variance within acceptable range
• Cases where rollback itself would cause different data loss (evaluate carefully)

Rollback Decision Tree

Error rate > SLO?
├── Yes → Can we fix forward in < 15 minutes? → No  → ROLLBACK
│                                              → Yes → hotfix + monitor
└── No  → Monitor, do not rollback

Rollback Commands

# Kubernetes rolling update — undo last rollout
kubectl rollout undo deployment/payment-service

# Kubernetes — target a specific revision
kubectl rollout undo deployment/payment-service --to-revision=3

# Argo Rollouts canary — abort and revert to stable
kubectl argo rollouts abort payment-service

# Helm — rollback to a previous release number
helm rollback payment-service 3

Rollback Runbook Template

## Rollback: [Service Name]

**Trigger criteria:** [e.g., error rate > 1% for 5 minutes]

**Steps:**
1. Notify on-call channel: "@oncall rolling back payment-service due to [reason]"
2. Run: `kubectl rollout undo deployment/payment-service -n production`
3. Verify: `kubectl rollout status deployment/payment-service`
4. Check metrics: confirm error rate returns to baseline
5. Create incident ticket with timeline and root cause

**Data rollback:** [specify if DB migration rollback is needed and how]
**Escalation:** [who to page if rollback fails]

See also: ci-cd, containerization, observability, incident-response

Red Flags

• Deploying a schema migration and an app change in the same atomic release — if the migration succeeds but the app rollout fails mid-way, old pods still running see the new schema; migrations and app deploys must be sequenced across separate releases
• Setting maxUnavailable: 1 instead of 0 for critical services — during a rolling deploy, one pod is taken down before the new one is ready, briefly dropping capacity below the desired replica count and increasing error rates
• Feature flag with no documented cleanup date — flags that ship but never get cleaned up accumulate into untested conditional branches; enforce a sprint deadline at the time of flag creation
• Blue/green flip without traffic warming on the Green environment — an un-warmed JVM or cold connection pool on Green produces a latency spike immediately after the flip that looks like an outage
• Canary rollback based only on error rate, ignoring latency SLO — a new version can stay under 1% errors while p99 latency doubles; always gate canary promotion on both error rate and latency thresholds
• Defining rollback criteria only after an incident starts — ad-hoc rollback decisions under pressure are slow and inconsistent; criteria and commands must be written in the runbook before the deploy
• Rolling back a migration by dropping a column that the old app version still reads — the old app immediately errors after the column is dropped; contract phases must be fully completed before any column is removed
• Using environment variables as a feature flag substitute for runtime toggles — env var flags require a pod restart to take effect and cannot be changed per-user or per-percentage; use a proper feature flag service for runtime control

Checklist

• Deployment strategy chosen and documented (rolling / blue-green / canary)
• maxUnavailable: 0 set for zero-downtime rolling updates
• Readiness probe passes before traffic is routed to new pods
• Smoke tests run automatically after each deployment
• Canary analysis configured with SLO-based pass/fail criteria
• Feature flags used for high-risk features — code deployed dark before enabling
• Dead feature flag code cleaned up within same sprint as full rollout
• Database migrations follow expand-contract pattern for zero-downtime
• Both app versions compatible with same DB schema during rolling deploy window
• Rollback procedure documented with specific commands and trigger criteria