agentsclimarketplace
Sign in

Deterministic coverage verification

Skill kjuhwa/skills-hub/skills/testing/deterministic-coverage-verification

Run LLVM-instrumented tests multiple times and verify identical coverage reports — catches compiler/env non-determinism that can mask real bugs.From its SKILL.md

Install
npx -y skills add kjuhwa/skills-hub --skill deterministic-coverage-verification

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.9 KB, 350 tokens by cl100k_base, as published. Nobody here has run it

Deterministic Coverage Verification

When to use

Any C/C++ project where test coverage is tracked in CI. Non-deterministic coverage hides flaky tests and reports false gains after unrelated changes.

Procedure

  1. Build the test binary with -fprofile-instr-generate -fcoverage-mapping (Clang) — no optimizer surprises in Debug.
  2. Run the full suite twice, each time with a distinct LLVM_PROFILE_FILE=<run>.profraw.
  3. Merge each run: llvm-profdata merge -sparse <run>.profraw -o <run>.profdata.
  4. Generate a line-level report per run: llvm-cov export -format=lcov <bin> -instr-profile=<run>.profdata > <run>.lcov.
  5. Diff the two lcov files. Any non-empty diff is a determinism failure — investigate before trusting the coverage number.

Common non-determinism sources

  • Thread scheduling in tests that sample std::chrono or std::thread::id.
  • Unseeded RNGs in fixtures.
  • Filesystem iteration order (glob results without sort).
  • -ffast-math / LTO changing branch layout.

Notes

  • Wire this as a nightly CI job, not per-PR — it's a smoke signal, not a gate.
  • For fuzz targets, the equivalent gate is "same corpus → same coverage bitmap" (see fuzz-corpus-seed-management).

Evidence

  • contrib/devtools/deterministic-fuzz-coverage/
  • contrib/devtools/deterministic-unittest-coverage/
  • contrib/devtools/README.md

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.

Keep looking

Skills are one crate of 326,835. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.