agentsclimarketplace

Test flakiness triage

Skill Raishin/vanguard-frontier-agentic/skills/qa/test-flakiness-triage

Use this skill when triaging flaky tests across any test framework — analyzing a flaky-test report, CI rerun history, or a set of intermittently failing tests to assign each a root-cause category and a remediation or quarantine path. Trigger when a user reports tests that pass on re-run, asks why CI is unreliable, provides a flaky-test dashboard export, or wants a quarantine policy. This skill reviews evidence statically; it does not execute or re-run tests.From its SKILL.md

Install
npx -y skills add Raishin/vanguard-frontier-agentic --skill test-flakiness-triage

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

3.8 KB, 697 tokens by cl100k_base, as published. Nobody here has run it

Test Flakiness Triage

Purpose

This skill triages flaky tests — tests that pass and fail without a code change — into root-cause categories and assigns each a remediation or quarantine path. Flakiness is a process failure, not a test-by-test annoyance: once a suite flakes, engineers re-run red builds reflexively, a re-run becomes the default response to any failure, and the suite stops detecting real regressions. The skill is framework-agnostic (Playwright, Cypress, Jest, JUnit, pytest, Go) and converts a pile of "sometimes fails" into categorized causes, a quarantine decision per test, and a policy that stops new flakes from entering the suite.

Lean operating rules

  • Triage every flaky test into exactly one primary root-cause category: async/timing race, test interdependence (shared state / order), environment coupling (clock, timezone, locale, network), non-deterministic data (random, current date, unseeded faker), resource contention (ports, DB, parallelism), or external dependency (third-party API, un-mocked network).
  • Treat a flaky test left in the gating suite with no quarantine and no owner as HIGH — every run it costs the whole team a re-run and erodes signal.
  • Treat "re-run until green" CI configuration (automatic unlimited retries, || true, retry-the-whole-job) with no flaky tracking as HIGH — it converts flakiness from visible to invisible and unbounded.
  • Treat a flaky test "fixed" only by adding a sleep, raising a timeout, or adding a retry as HIGH — the root cause is still present and will resurface.
  • Treat tests sharing a mutable fixture, database row, file, or port without per-test isolation as HIGH for interdependence flakiness.
  • Treat assertions on wall-clock time, Date.now, timezone, or locale without injection/freezing as HIGH for environment coupling.
  • Treat un-mocked calls to third-party services in a unit or component test as HIGH — network and rate limits make them flaky by construction.
  • Treat a quarantine with no expiry, no owner, and no tracking issue as MEDIUM — quarantine is a holding cell, not a graveyard; without an exit it becomes permanent coverage loss.
  • Treat the absence of any flaky-detection mechanism (no rerun-diff, no flaky reporter, no quarantine list) as MEDIUM — flakiness is then discovered only by frustrated engineers.
  • Do not recommend deleting a flaky test as the default fix — a flaky test usually covers a real path; quarantine with an owner and a fix deadline preserves the coverage intent.
  • Label every finding with evidence basis: failure logs / rerun history provided, test source provided, documentation-based, or inference.

References

Load these only when needed:

Response minimum

Return, at minimum:

  • Per-test root-cause category assignment
  • Quarantine decision per test (quarantine now / fix in place / keep gating)
  • Root-cause remediation direction per category
  • Quarantine policy assessment (owner, expiry, tracking)
  • CI retry-configuration findings
  • Severity-labelled finding list (critical / high / medium / low)
  • Safe next actions

What ships with it: 2 files

6.6 KB alongside SKILL.md

references/

Keep looking

Skills are one crate of 613,531. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.