Flaker management
Operate @mizchi/flaker after setup. Use when the user asks how to run flaker day-to-day, review sampling and flaky metrics, design advisory vs required CI gates, promote or demote Playwright E2E or VRT checks, tune PR time budgets, run nightly triage, or manage quarantine and `@flaky` tags in an OSS repository. Targets @mizchi/flaker 0.7.0+ (declarative apply model).From its SKILL.md
npx -y skills add mizchi/flaker --skill flaker-managementAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 24 stars24 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
6.3 KB, as published. Nobody here has run it
flaker management skill
flaker-management is the operational companion to flaker-setup.
flaker-setupInstall, initialize, and wire the first advisory lane.flaker-managementRun the lane over time, review health via drift, promote or demote checks, and keep flaky tests from eroding trust.
If the repository does not have flaker.toml and no CI lane yet, use flaker-setup first.
When this skill applies
- "flaker の運用方法を決めたい"
- "advisory から required にいつ上げるべき?"
- "E2E VRT を段階的に gate に入れたい"
- "nightly で flaky を triage したい"
- "quarantine をどう回す?"
- "週次レビューの playbook を作りたい"
Mental model: apply + drift
flaker.tomlis the desired state (gates, profiles,[promotion]thresholds,[quarantine].auto).flaker applyis the reconciler — idempotent; safe to run hourly/daily/on-demand. It auto-runscollect/calibrate/quarantine applyas needed based on current DB state.flaker statusis the drift detector — reports which[promotion]thresholds are unmet, so promotion readiness is a boolean (ready/not ready), not a judgement call.
The canonical daily loop is:
flaker apply && flaker status
Read order
- Read
../../docs/operations-guide.ja.mdor../../docs/operations-guide.mdfirst, depending on the user's language. - Read
../../docs/flaker-management-quickstart.ja.mdor../../docs/flaker-management-quickstart.mdfor the first 10 minutes. - Read
references/management-guide.ja.mdfor the full operating model. - If the user wants theory or justification, read
references/theory.ja.md. - If the user wants copy-paste defaults, read
references/presets.ja.md. - Reuse templates from
assets/instead of rewriting them.
What to inspect first
flaker.toml— especially the[promotion]thresholds (defaults are documented; overriding signals intent)- current GitHub Actions topology:
pull_request,push,schedule - latest
flaker statusoutput (drift + activity + health in one page) flaker status --gate merge --detail --jsonwhen you need exact promotion metricsflaker ops weeklyfor quarantine / flaky trend bundles- whether
@flakytagging or quarantine manifest is already in use - current PR runtime budget
- whether the focus is generic CI health, or specifically Playwright E2E / VRT
Required output shape
When applying this skill, return:
- lane design:
learning/verdict/rebalance - promotion criteria (align with
[promotion]inflaker.toml; override only with justification) - demotion criteria
- review cadence: per-PR / daily / weekly (daily is usually just
flaker apply && flaker status) - exact
flakercommands, config, and workflow snippets
Guardrails
- Do not move new E2E / VRT checks straight into required CI.
- Do not treat retries as proof of stability.
- Do not let quarantine become a graveyard; attach an owner and an exit rule.
- Keep a full scheduled lane even after PR gating starts.
- For AI-generated code, require a short per-test contract so visual checks encode intent, not just pixels.
- Do not promote
--gate mergeto required untilflaker statusdrift reportsready. - Do not use deprecated aliases in new scripts —
analyze kpi,analyze eval,collect ci,debug doctor,quarantine suggest/apply,gate review/history/explainall print deprecation warnings in 0.7.0 and will be removed in 0.8.0. Use the primary commands instead.
flaker commands to prefer
# Daily
flaker apply
flaker status
# Weekly operator review
flaker status --markdown > .artifacts/status-weekly.md
flaker ops weekly --output .artifacts/flaker-weekly.md
flaker status --gate merge --detail --json > .artifacts/merge-gate.json
# Promotion snapshot (authoritative metrics)
flaker gate review merge --json > .artifacts/gate-review-merge.json # DEPRECATED in 0.7.0; use status --gate merge --detail --json
# Incident
flaker debug retry
flaker debug confirm "<suite>:<test>" --repeat 10
flaker debug bisect --test "<name>"
Note: ops daily / weekly / incident are still first-class primary commands — apply does NOT emit the daily artifact yet. Use them directly.
Promotion / demotion decision rule
Promote --gate merge advisory → required iff flaker status drift reports ready (all 5 [promotion] thresholds met). Primary signal is flaker status — the drift section shows ready or lists unmet thresholds:
matched_commits ≥ [promotion].matched_commits_min(default 20)false_negative_rate ≤ [promotion].false_negative_rate_max_percentage(default 5%)pass_correlation ≥ [promotion].pass_correlation_min_percentage(default 95%)holdout_fnr ≤ [promotion].holdout_fnr_max_percentage(default 10%)data_confidence ≥ [promotion].data_confidence_min(defaultmoderate)
Demote back to advisory when ANY of the following holds for 1+ week:
- unexplained false failures continue
- flaky count trend rises and erodes trust
- owner becomes unavailable
- runtime budget is exceeded
Anti-patterns
- Using raw
flaker collect ci/flaker collect calibrate(deprecated in 0.7.0) in daily cron whenflaker applyalready handles the ordering and idempotency. - Using
flaker analyze kpi(deprecated) instead offlaker status, orflaker analyze eval --markdown(deprecated) instead offlaker status --markdown. - Basing promotion on
flaker statusnumbers alone when they look close —flaker status --gate merge --detail --jsonis the authoritative source for exact values (the deprecatedflaker gate review merge --jsonform also still works with a stderr warning). - Ignoring
flaker statusdriftholdout_fnrwhenholdout_ratio = 0; if holdout isn't configured, the threshold cannot be evaluated and drift treats it as unmet. Either configure[sampling].holdout_ratioor accept that holdout FNR will gate promotion.