agentsclimarketplace

Test flakiness triage

Skill Raishin/vanguard-frontier-agentic/skills/qa/test-flakiness-triage

Curated marketplace of AI skills, agents, and rules for cloud, zero-trust, and compliance-aware engineering - works with Claude Code, Codex, Cursor, Copilot, and more.

Install
npx -y skills add Raishin/vanguard-frontier-agentic --skill test-flakiness-triage

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 18 stars18 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Use this skill when triaging flaky tests across any test framework — analyzing a flaky-test report, CI rerun history, or a set of intermittently failing tests to assign each a root-cause category and a remediation or quarantine path. Trigger when a user reports tests that pass on re-run, asks why CI is unreliable, provides a flaky-test dashboard export, or wants a quarantine policy. This skill reviews evidence statically; it does not execute or re-run tests.

SKILL.md

3.8 KB, 697 tokens by cl100k_base, as published. Nobody here has run it

Test Flakiness Triage

Purpose

This skill triages flaky tests — tests that pass and fail without a code change — into root-cause categories and assigns each a remediation or quarantine path. Flakiness is a process failure, not a test-by-test annoyance: once a suite flakes, engineers re-run red builds reflexively, a re-run becomes the default response to any failure, and the suite stops detecting real regressions. The skill is framework-agnostic (Playwright, Cypress, Jest, JUnit, pytest, Go) and converts a pile of "sometimes fails" into categorized causes, a quarantine decision per test, and a policy that stops new flakes from entering the suite.

Lean operating rules

  • Triage every flaky test into exactly one primary root-cause category: async/timing race, test interdependence (shared state / order), environment coupling (clock, timezone, locale, network), non-deterministic data (random, current date, unseeded faker), resource contention (ports, DB, parallelism), or external dependency (third-party API, un-mocked network).
  • Treat a flaky test left in the gating suite with no quarantine and no owner as HIGH — every run it costs the whole team a re-run and erodes signal.
  • Treat "re-run until green" CI configuration (automatic unlimited retries, || true, retry-the-whole-job) with no flaky tracking as HIGH — it converts flakiness from visible to invisible and unbounded.
  • Treat a flaky test "fixed" only by adding a sleep, raising a timeout, or adding a retry as HIGH — the root cause is still present and will resurface.
  • Treat tests sharing a mutable fixture, database row, file, or port without per-test isolation as HIGH for interdependence flakiness.
  • Treat assertions on wall-clock time, Date.now, timezone, or locale without injection/freezing as HIGH for environment coupling.
  • Treat un-mocked calls to third-party services in a unit or component test as HIGH — network and rate limits make them flaky by construction.
  • Treat a quarantine with no expiry, no owner, and no tracking issue as MEDIUM — quarantine is a holding cell, not a graveyard; without an exit it becomes permanent coverage loss.
  • Treat the absence of any flaky-detection mechanism (no rerun-diff, no flaky reporter, no quarantine list) as MEDIUM — flakiness is then discovered only by frustrated engineers.
  • Do not recommend deleting a flaky test as the default fix — a flaky test usually covers a real path; quarantine with an owner and a fix deadline preserves the coverage intent.
  • Label every finding with evidence basis: failure logs / rerun history provided, test source provided, documentation-based, or inference.

References

Load these only when needed:

Response minimum

Return, at minimum:

  • Per-test root-cause category assignment
  • Quarantine decision per test (quarantine now / fix in place / keep gating)
  • Root-cause remediation direction per category
  • Quarantine policy assessment (owner, expiry, tracking)
  • CI retry-configuration findings
  • Severity-labelled finding list (critical / high / medium / low)
  • Safe next actions

Keep looking

Skills are one crate of 328,083. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.