Flaky test audit

Flaky Test Audit

Purpose

Restore trust in the test suite by eliminating flakiness at its source: identify tests that pass and fail without code changes, diagnose the real cause, and fix it — because a flaky suite gets ignored, then bypassed, then guards nothing.

When to Use

• When tests fail intermittently, CI needs reruns to go green, or people say "just re-run it."
• Proactively before flakiness normalizes bypassing.
• Not for genuinely-failing tests (those are real bugs — ../../bug-investigation).

Inputs

• CI history / rerun logs showing intermittent failures.
• The flaky tests + their environment (test-environment-management) and data (test-data-management).

Discovery Questions

• Which tests fail without related code changes, and how often?
• Does the failure depend on order, parallelism, timing, or wall-clock/date?
• Does it touch shared state, real time/randomness, or a live external?

Responsibilities

• Detect: mine CI for pass/fail-without-change signals; rank by frequency and by what they guard.
• Diagnose the root cause class:
  • timing (fixed sleeps, races, missing awaits) → deterministic waits / auto-wait (playwright-e2e/maestro-e2e);
  • shared/leaked state or order dependence → per-test isolation (test-data-management);
  • real clock/randomness → fake timers, seeded randomness;
  • live externals / network → fakes at interfaces (test-environment-management);
  • environment nondeterminism (fonts, viewport, locale) → pin it (visual-regression).
• Fix the cause, not the symptom: no blanket retries, no sleep bumps, no .skip as the resolution.
• Quarantine sparingly and temporarily: a persistently-flaky test may be isolated from the gate with a tracked ticket and an owner, never silently disabled — and fixed or deleted promptly.
• Add guardrails: run tests in random order in CI to surface order dependence; treat new flakiness as a same-day issue.

Required Workflow

1. Detect flaky tests from CI history; rank by impact.
2. Reproduce (repeat runs, randomized order, parallelism).
3. Diagnose the root-cause class.
4. Fix the cause at the right layer.
5. Quarantine only if unavoidable — ticketed, owned, time-boxed; verify stability post-fix.

Decision Rules

• Fix the root cause; retries and sleep-bumps hide flakiness and let it spread.
• Randomized-order CI runs are the cheapest way to expose order dependence — enable them.
• Quarantine is a temporary bridge with a ticket, not a resting place; a skipped test guards nothing.
• A flaky test that can't be made deterministic is a candidate for deletion, not permanent retry.

Rules

• No blanket auto-retries masking flakiness.
• Quarantined tests are tracked, owned, and time-boxed.
• New flakiness is treated as a real defect, not noise.

Anti-Patterns

• "Just re-run CI" as the standard response.
• Adding waitForTimeout/sleeps to stabilize timing.
• .skip as the permanent fix.
• Global test retries hiding real races.
• Ignoring order dependence by always running in the same order.

Validation Checklist

• Flaky tests detected + ranked from CI history.
• Root-cause class diagnosed per test.
• Cause fixed at the right layer (no retry/sleep bandaids).
• Any quarantine is ticketed, owned, time-boxed.
• Randomized-order runs enabled; stability verified.

Definition of Done

Flaky tests identified, root-caused, and fixed at the source — with randomized-order CI guarding against regression and any quarantine tracked and time-boxed — so the suite is trusted again.

Related Skills

test-data-management, test-environment-management, playwright-e2e, maestro-e2e, visual-regression, test-coverage-audit, ../../bug-investigation, ../../devops/ci-cd.

Related Knowledge

../../../knowledge/ (known flaky areas, CI history).

Related References

../../../references/testing/ (flakiness patterns, when populated).

Context Loading Guidance

• Requires: CI failure history, the flaky tests + their env/data.
• Does not require: the whole suite, unrelated passing tests.
• May load: test-data-management, test-environment-management.
• Stop when: causes are fixed and the suite is stable under randomized order.

Token Efficiency Guidance

Rank by impact and work the top offenders; the flaky-test → cause → fix table is the artifact, not a full-suite review.