Deflake it

Skill ravidsrk/orca-fleet/skills/deflake-it

Eliminate intermittent test failures to a declared confidence contract. Detect flakes by repeat-run + CI retry history, diagnose each with a loop that RAISES the failure rate (never a theory), fix the root cause and ratchet it red-by-revert, then re-run the whole suite for a consecutive green streak — local AND in CI — looping until the streak holds. Retry-wrappers as fixes are banned and grep-checked. Use when "kill the flaky tests", "deflake the CI", "flake zero", a flaky/intermittent suite. Not for a single hard bug with no suite-rate contract (root-cause) or deterministic failures (clean-sweep).From its SKILL.md

Install

npx -y skills add ravidsrk/orca-fleet --skill deflake-it

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as MIT. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

5.3 KB, as published. Nobody here has run it

deflake-it — green N times in a row, local and CI

You are the COORDINATOR. The unit is an INTERMITTENT FAILURE DISTRIBUTION, not a defect: detection and proof require REPEATED observations, environment correlation, and a consecutive-green / statistical contract. Rerun behavior IS the mission, not verification-after-a-fix. Composes diagnose, build-change, remediate-finding (the finding is a flake; the red-by-revert ratchet is its negative control), acceptance-review, compound-learn; rides merge-serialization, reviewed-sha-freshness, dispatch-lifecycle, liveness-resume, evidence-manifest, ledger-contract, attention-budget. The suite green streak IS the prove step (no separate runtime-prove pass). Worker TASK pack: one of matt | addy.

Two terminal outcomes

STABLE — the full suite passes GREEN_STREAK consecutive runs, local AND CI, with zero flakes and zero retry-wrappers. The streak is a PRE-DECLARED statistical contract, not a vibe: N green runs bound the residual per-run flake rate at p ≤ 1 − 0.05^(1/N) with 95% confidence (N=10 only proves p ≲ 26%; N=30 → p ≲ 9.5%; N=100 → p ≲ 3%; catching a 1% flake needs ~300). The bound assumes INDEPENDENT runs — correlated flakes (shared state, time-of-day, load) violate it, so vary seed/order/parallelism/time across the streak and treat the bound as optimistic. Default: declare target residual rate, derive N (default GREEN_STREAK=30 if unspecified). Record both target rate and N in the manifest's metric_contract.
STABLE-WITH-QUARANTINE (degraded) — ≥1 flake survives diagnosis with no root cause and is quarantined with a human-approved tracking ticket. Never reported as STABLE.

Pipeline

DETECT: run the suite DETECT_RUNS times (default 30; parallel, varied seed/order) → per-test flake RATE;
  mine CI retry history (pass-on-retry tests flake in an env local runs don't reproduce — capture even
  at local rate 0). Deterministic N/N failures are BUGS → route to clean-sweep, out of scope.
  → DIAGNOSE (diagnose playbook, adapted): build a loop that RAISES the failure rate (tight loop, under
    load, clock skew, shuffled order, shared-state siblings); classify the taxonomy (order-dependence /
    shared mutable state / real time-or-tz / network / unseeded RNG / resource leak / too-tight timeout
    / async race) → the class dictates the fix.
  → BOOTSTRAP integration BASE (runtime/scripts/preflight.py --base <BASE> --fork-point <sha
    recorded in the ledger header at BASE creation>; BASE ≠ default — dispatch-lifecycle.md)
  → FIX root cause + RATCHET red-by-revert (revert only the fix, show the flake returns at its measured
    rate; restore, show it's gone across a mini-streak) — never `retry(3)` a flake into hiding.
  → close per remediate-finding: PR-per-flake against BASE → build-blind REVIEW (acceptance-review)
    → conductor LAND
  → PROVE: full suite GREEN_STREAK consecutive runs, local AND verified in CI (gh run list). ANY
    flake resets the streak to zero and re-enters detection. → loop → outcome → REFLECT (`compound-learn`)

Convergence proof

Every detected flake (local AND CI-only) reaches a terminal state: root-caused + fixed + merged (with a red-by-revert ratchet) OR quarantined with a human-approved ticket — no "documented and left flaky" exit. CI-only flakes: the local streak does NOT disprove them; each fixed-and-verified-in-CI (green across the same resolved_N consecutive CI triggers, gh run list pasted) or quarantined. The streak (timestamps + seeds/orders per run) pasted, local AND CI. Zero retry/rerun wrappers added (grep the diff). Manifest names STABLE or STABLE-WITH-QUARANTINE.

Ledger + supervision

Header per liveness-resume.md: RUN · COORDINATOR · BASE · FORK_POINT · T0 · SOURCE · WIP (- if N/A; SOURCE = DETECT_RUNS · GREEN_STREAK=resolved_N · target residual rate). Rows include Orca task id + flake fields. Stalls → WATCH; death → RESUME scoped to header coordinator + ledger task ids, git-verified.

Anti-patterns

Theorizing before a loop that reproduces at elevated rate. retry(n) / --rerun-failures as the fix (hides flakes, poisons the signal). One green run = done. Treating an N/N deterministic failure as a flake (it's a bug — clean-sweep). Widening a timeout that masks a real race.

root-cause (a single hard intermittent bug, not a suite-rate contract), prove-it (coverage), clean-sweep (deterministic bugs).