Testing in production

<objective> Production is the only environment that is production. Every other environment is an approximation — staging never replicates real data volume, traffic, or third-party quirks. This skill covers how to validate quality in production safely: controlled blast radius, automated rollback, guardrail metrics, and smoke tests that catch problems before users do. The recurring failure it prevents: shipping to 100% of users with no flag to flip, no baseline to compare against, and no tested way back. </objective>

Quick Route

Situation	Go to
Shipping a feature behind a flag	Feature Flag Testing
Ramping traffic 1% → 100% with gates	Progressive Rollout + references/rollout-policy.md
Need post-deploy checks on every release	Production Smoke Tests + references/patterns.md
Deciding what numbers gate the rollout	Guardrail Metrics
New code path with no user-visible change yet	Dark Launches
Proving the rollback actually works	Verification

---

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there.

Feature flag system:

• Do you have a feature flag platform? (LaunchDarkly, Statsig — now part of OpenAI, GrowthBook, Unleash, Flagsmith, Harness FME — formerly Split, custom, none)
• How are flags managed? (Dashboard, config file, environment variables)
• Can flags target specific users, percentages, or segments?
• How many active flags exist today? Is there a cleanup process?

Rollout capability:

• Can you deploy to a subset of traffic? (Canary infrastructure, weighted routing, feature flags)
• How long does a deployment take? How long does a rollback take?
• Do you have blue-green or rolling deployments?
• Can you route traffic by region, user cohort, or percentage?

Monitoring maturity:

• What observability is in place? (APM, logging, error tracking, metrics)
• Do you have dashboards for error rate, latency, and business metrics?
• Are alerts configured with appropriate thresholds?
• Can you compare metrics between canary and baseline in real time?

Production access and safety:

• Who has production access? Is there an approval process?
• Are there dedicated test accounts in production?
• Can you run operations in production without affecting real user data?
• Is there a production incident response process?

---

Core Principles

1. Production is the final test environment

Staging approximates production. It does not replicate production's data volume, traffic patterns, third-party integrations, infrastructure quirks, or user behavior. Testing in production is not reckless — it is realistic. The question is not whether to test in production, but how to do it safely.

2. Safety through blast radius control

Every production test must answer: "If this goes wrong, how many users are affected?" The answer must be as small as possible. Feature flags, canary deploys, and traffic splitting exist to shrink the blast radius from 100% to 1% or less.

3. Always have a tested rollback plan

Before any production test begins, the rollback mechanism must be identified, tested, and fast. "Disable the flag" is a good rollback plan. "Redeploy the previous version" is acceptable. "We'll figure it out" is not a plan. A rollback you have never fired is a hypothesis, not a plan — see Verification for how to prove it works.

4. Monitoring is a prerequisite, not a nice-to-have

You cannot test in production without monitoring. If you cannot measure error rates, latency, and business metrics in real time, you cannot detect problems. Fix monitoring gaps before adding production tests.

5. Production tests must be non-destructive

Production tests must never corrupt real user data, send real notifications to real users, charge real payment methods, or create side effects that require manual cleanup. Synthetic accounts, test flags, and isolated resources are mandatory.

---

Feature Flag Testing

Feature flags are the safest mechanism for production testing. They decouple deployment from release and provide instant rollback.

Test with flags ON and OFF

Every flagged feature needs tests in both states. The flag-off path is the rollback path and must work flawlessly — use setFeatureFlag(name, true|false, { userId: TEST_USER_ID }) to drive both. See references/patterns.md for the full ON/OFF test pair.

Flag lifecycle testing

Flags are not just on or off. They transition through states, and each transition must be validated.

Flag lifecycle:
  Created → Targeting internal users → Canary (1%) → Partial (10-50%) → Full (100%) → Cleanup (removed)

Test at each stage:
  - Internal: Feature works for internal accounts, hidden from external
  - Canary: Metrics are comparable between flag-on and flag-off cohorts
  - Partial: No performance degradation at scale
  - Full: All user segments work correctly
  - Cleanup: Code with flag removed behaves identically to flag-on

Stale flag cleanup

Flags left in code become technical debt. Run a weekly CI job that queries the flag provider for flags that are 100% rolled out and older than 14 days. These are candidates for code cleanup — remove the flag branching logic and retain only the enabled path. Removing the flag without removing the dead code is half a cleanup.

Flag combination testing

When multiple flags interact, test the combinations that matter. Do not test all 2^N combinations — focus on flags that affect the same user flow (e.g. new-checkout, express-pay, discount-engine-v2 all touch checkout). Pick the critical combinations: all-new, a representative mixed state, and all-legacy. See references/patterns.md for the combination-test loop.

---

Progressive Rollout

Vendor-native canary analysis. Before hand-rolling the rollout-policy YAML below, check whether your platform already does it: LaunchDarkly Guarded Rollouts (auto-monitored progressive rollouts with metric-based auto-rollback; uses a frequentist sequential-testing analysis model since early 2026), Statsig Auto-tune, Argo Rollouts AnalysisRun, Flagger, Harness Continuous Verification. If you have one, prefer it — the integration with your metrics and rollback mechanics is cheaper than maintaining a custom analysis loop.

AI feature rollout is its own pattern: model variant + prompt as a flag value, with cost guardrails and a kill switch. LaunchDarkly AI Configs / AgentControl (AI Configs rebranded under the AgentControl umbrella, announced May 2026) is the documented path for shipping LLM features behind progressive rollout. See release-readiness for the full pattern.

Canary stages: 1% to 100%

A structured rollout with explicit promotion criteria at each stage.

Stage	Traffic	Hold Time	Key Checks
Canary	1%	15-30 min	Error rate, crash rate, exceptions
Early adopters	10%	1-2 hours	Latency P95, conversion rate
Partial	50%	2-4 hours	All guardrails, business metrics
Full	100%	24 hours monitoring	Long-tail issues, batch job compatibility

Automated promotion and rollback

Define machine-checkable conditions for advancing between stages — hold_duration plus metric conditions (error_rate_5xx < 0.5%, latency_p95 < 500ms, crash_rate == 0). Automatic rollback fires when guardrails are breached, with no human approval needed: error_rate_5xx > 2x_baseline for 5m, latency_p99 > 3x_baseline for 5m, crash_rate > 0.1% for 2m, each notifying on-call. Gate on error-budget burn rate, not only raw multipliers, so slow burns that still blow the SLO are caught. See references/rollout-policy.md for the full promotion YAML, rollback triggers, and SLO-gate config.

Verify the rollback fired correctly

Auto-rollback firing is not the same as the incident being resolved. After a rollback triggers, do not declare "recovered" until you have confirmed the system is actually back:

1. Re-run the health-check smoke test against production (GET /api/health returns healthy and the previous version).
2. Confirm the flag/deploy state reverted — query the flag platform that the flag is off (or the deploy that the previous build is serving), don't assume.
3. Confirm guardrail metrics returned to baseline — error rate, P99 latency, and crash rate back within their pre-deploy windows.
4. Confirm the rollback notification reached on-call (Slack/PagerDuty) so the incident is owned.

Only when all four hold is the rollback verified. See Verification for the staging dry-run that proves this chain before first production use.

---

Production Smoke Tests

Post-deploy critical path tests

Run immediately after every deployment as a pipeline stage (not only in pre-deploy CI). These verify core functionality works with production configuration, data, and infrastructure: a /api/health check, the authentication flow with synthetic credentials, core data loading, and search. Configure retries: 1 and a timeout so a flaky post-deploy run doesn't block the pipeline on the first blip. See references/patterns.md for the full production-smoke.spec.ts.

Synthetic user accounts

Production test accounts must be clearly distinguishable from real users: a reserved email pattern (smoke-test+{env}@yourcompany.com), an is_synthetic = true flag, and exclusion from analytics, billing, and email campaigns. Create them via admin API, and prefer short-lived OIDC / workload-identity tokens over long-lived passwords. See references/patterns.md for the full conventions.

Non-destructive assertions

Production smoke tests must read, not write. When writes are unavoidable, clean up in fixture teardown so cleanup runs whether the test passes or fails.

Do not call test.afterEach() inside a test() body — Playwright registers hooks at describe/file scope, so a hook registered mid-test never schedules teardown and throws test.afterEach() can only be called in a describe block. The data leaks. Use an auto-cleanup fixture that records created resource IDs and deletes them on teardown (or try/finally for a one-off script). See references/patterns.md for the fixture-based create-verify-cleanup pattern.

---

Guardrail Metrics

What to monitor during rollout

Category	Metric	Comparison Method	Alert Threshold
Errors	HTTP 5xx rate	vs. pre-deploy baseline	>2x baseline for 5 min
Errors	Unhandled exception count	vs. pre-deploy baseline	Any new exception type
Latency	P50 response time	vs. pre-deploy baseline	>1.5x baseline
Latency	P95 response time	vs. pre-deploy baseline	>2x baseline
Latency	P99 response time	vs. pre-deploy baseline	>3x baseline
Business	Conversion rate	vs. 7-day average	Drop >5%
Business	Revenue per session	vs. 7-day average	Drop >10%
Client	Crash rate (mobile)	vs. previous release	>0.1% increase
Client	JavaScript error rate	vs. pre-deploy baseline	>2x baseline
Infra	CPU utilization	absolute	>80% sustained
Infra	Memory utilization	absolute	>85% sustained

Baseline comparison

Compare canary metrics against a control group running the previous version, not against historical data alone.

Comparison approaches (best to worst):
  1. Canary vs. control: split traffic, compare groups in real time (best)
  2. Before/after: compare post-deploy metrics to pre-deploy window (good)
  3. Historical: compare to same time last week (acceptable for trends)
  4. Absolute thresholds: fixed thresholds regardless of baseline (fragile)

Statistical significance

For business metrics (conversion, revenue), small sample sizes produce noisy results. Wait for statistical significance before drawing conclusions.

Minimum sample sizes for rollout decisions:
  - Error rate: 1,000 requests (errors are rare events, need volume)
  - Latency: 500 requests (more stable, converges faster)
  - Conversion rate: 5,000 sessions (business metrics have high variance)
  - Crash rate: 10,000 app launches (crashes are rare events)

Rule of thumb: if you don't have enough traffic at 1% to reach
significance in 30 minutes, increase to 5% or extend the hold window.

---

Dark Launches

Dark launches deploy new functionality to production but hide it from users. Real production traffic exercises the new code path without user-visible impact.

Traffic shadowing

Duplicate incoming requests to the new service. Compare responses without returning the new response to the user.

Request flow:
  User → Load Balancer → Production Service (returns response to user)
                       ↘ Shadow Service (processes request, logs result, discards)

What to compare:
  - Response status codes: shadow should match production
  - Response body: diff for semantic equivalence (ignore timestamps, IDs)
  - Latency: shadow should not be significantly slower
  - Error rate: shadow should not produce more errors

Parallel execution

For migrations (new database, new algorithm, new service), run both the old and new path in production. The old path returns the result to the user; the new path runs asynchronously, logs differences, and discards its result. Track the match rate over time — target 99%+ match before cutting over.

Shadow launch timeline:
  Week 1: Deploy shadow, start comparing, expect <50% match
  Week 2: Fix mismatches, match rate should climb to 90%+
  Week 3: Match rate stable at 99%+, handle remaining edge cases
  Week 4: Cut over: shadow becomes primary, old becomes shadow
  Week 5: Remove old path after 1 week of stability

---

Anti-Patterns

Testing in production without monitoring

Running production tests without dashboards and alerts is flying blind. You will not know if your tests caused an issue until a user reports it. Fix: Monitoring is a prerequisite. Before adding any production test, verify you can see error rates, latency, and key business metrics in real time. Set up alerts before the first test runs.

No rollback plan

"We'll deploy a fix if something goes wrong" is not a rollback plan. Under pressure, fixes take longer, introduce new bugs, and extend the outage. Fix: Every production test or rollout must have a documented rollback mechanism that takes less than 5 minutes to execute — feature flag disable, previous deployment, or traffic reroute — and a verification step that confirms it actually recovered the system (see Verification).

Destructive operations in production tests

Production tests that create real orders, send real emails, or modify real user data are not tests — they are incidents waiting to happen. Fix: Use synthetic accounts flagged as test data. Use sandbox modes for payment and email. Clean up created data in fixture teardown so it always runs. If a test cannot be made non-destructive, it does not belong in production.

Cleanup that only runs on success

A cleanup step placed after an assertion never runs when the assertion fails, leaking exactly the test data it was meant to remove. Calling test.afterEach() inside a test() body is the same trap — it throws or is ignored. Fix: Put teardown in an auto-cleanup fixture or a finally block so it runs on pass and fail alike. See references/patterns.md.

Testing in production instead of pre-production

Production testing supplements pre-production testing. It does not replace it. If staging is broken and you are "testing in production" because it is the only working environment, fix staging first. Fix: Maintain a working pre-production environment. Use production testing for what only production can validate: real traffic, real data volumes, real third-party integrations.

Canary deploys without comparison

Deploying to 1% of traffic but not comparing canary metrics against a control group misses the entire point. You are just deploying slowly, not detecting problems. Fix: Always compare canary metrics against a baseline. Use side-by-side dashboards or automated canary analysis tools (Kayenta, Argo Rollouts analysis).

Stale feature flags

Flags that are fully rolled out but never removed accumulate. After a year, you have 200 flags with unknown interactions, and every code path has branching logic that nobody understands. Fix: Every flag gets an expiration date at creation time. After full rollout + 2 weeks of stability, remove the flag and its dead branch. Track flag age and alert when flags exceed their expiration.

---

Verification

Prove the rollback path actually fires before the first production use — a rollback you have never triggered is a hypothesis. Smallest check first:

1. Trip a guardrail in staging. Inject failure (e.g. force error_rate_5xx above 2x_baseline, or fail the health check 3 times) on a staged rollout wired to the same automatic_rollback policy. Confirm the rollback action fires within its for: window.
2. Confirm the reverted state. Re-run the health-check smoke test and assert status === 'healthy' and the previous version. Query the flag platform/deploy that the previous build is serving — don't assume.
3. Confirm metrics recovered. Error rate, P99 latency, and crash rate are back inside their pre-deploy windows.
4. Confirm the notification fired. The rollback alert reached the on-call channel (Slack/PagerDuty).
5. Smoke tests are wired into the pipeline. Run the deploy job against staging and confirm the post-deploy smoke stage executes and gates promotion — npx playwright test production-smoke.spec.ts exits 0.

If steps 1–4 cannot be demonstrated in staging, the rollback is unverified and the rollout is not ready.

Agent shortcut: vendor MCP servers exist for LaunchDarkly, GrowthBook, Unleash, Flagsmith, Statsig, and Harness FME, letting an AI agent flip flags and read rollout metrics directly during these checks rather than driving the dashboard by hand.

---

Done When

• Feature flag rollout plan is documented with explicit percentage steps (1% → 10% → 50% → 100%) and named guardrail metrics at each stage.
• Canary analysis is configured with automated pass/fail criteria so promotion and rollback decisions do not require manual metric comparison.
• Production smoke tests run as a pipeline stage on every deploy (not only in CI pre-deploy) and the stage exits 0 against production.
• Rollback trigger conditions are defined, documented, and demonstrated to fire correctly via the Verification staging dry-run (guardrail tripped → rollback fired → reverted state and recovered metrics confirmed) before first production use.
• Production test data strategy is documented, specifying whether synthetic users or anonymized real users are used and how they are excluded from analytics and billing.

Reference Files (in references/)

• rollout-policy.md — full promotion-criteria YAML, automatic-rollback triggers, and the error-budget / SLO-gate config.
• patterns.md — flag ON/OFF and combination tests, the production-smoke.spec.ts suite, synthetic-account conventions, and the fixture-based non-destructive create-verify-cleanup pattern.

Related Skills

• release-readiness — Go/no-go for the whole release; production testing is the post-deploy verification step inside it. Go there for the release checklist, not the rollout mechanics.
• synthetic-monitoring — Scheduled probes that run continuously after the rollout is complete. Go there for ongoing SLA validation, not in-flight rollout safety.
• observability-driven-testing — Uses prod traces and logs as the input to design new tests. Go there when telemetry tells you what to test, not when you need to ship safely.
• qa-metrics — Where guardrail metrics and rollout criteria feed into dashboards and KPIs.
• ci-cd-integration — Wiring the smoke-test stage and rollout gates into the pipeline.
• test-environments — Pre-production environments that production testing complements, never replaces.