Trustworthy tests

trustworthy-tests — a test must be able to fail for the right reason

Purpose: AI agents produce green test suites that prove nothing — tests that assert on the mocks they just set up, mirror the implementation line for line, and pass while the actual feature is broken because the agent never ran it. And they declare "done / production-ready" on confidence alone, which Stanford showed is inversely correlated with correctness. This skill makes tests load-bearing and makes "done" require evidence.

The deeper reason execution is non-negotiable: the most expensive slop is clean-but-wrong — idiomatic, well-structured code with an off-by-one, an inverted condition, a wrong default. It is invisible to on-sight review by construction; only running the code catches it. Every other craft skill reads the diff. This one runs it.

This is discipline, not mechanics. For the how of writing tests use the existing test-coverage-analyzer, pytest-patterns, jest-react-testing. This skill decides whether a test is worth anything and whether the work is actually finished.

The cardinal rule

A test you can't imagine failing is not a test. Before accepting any test, ask: "What real bug would make this go red?" If the only answer is "if my mock changes," it tests nothing.

Reflex card

1. Test observable behavior, not implementation. Assert on what the function returns or the state the user can see — not on which private methods got called in what order. Implementation-mirroring tests break on every refactor and catch no real bugs.
2. No tautological tests. A test whose only assertion is mock.assert_called() / expect(spy).toHaveBeenCalled() — with no check on the actual result — is theater. Mock the boundary, assert on the outcome.
3. Mock only true boundaries. Network, clock, randomness, third-party services. Don't mock the thing under test, and don't mock your own pure logic — call it for real (cheaper and more honest). Over-mocking is how green-but-meaningless suites happen.
4. Cover the edges, not just the happy path. The empty input, the boundary value, the error path, the malformed payload. The happy path is what already works in the demo; the edges are where production breaks.
5. Prove it ran (the evidence gate). Before saying "done," actually execute the code end-to-end and show the evidence: the test output, the curl request/response, the log line, the screenshot. "It should work" / "this is production-ready" without evidence is not done. Your confidence is near-zero signal.
6. Prove the test tests the fix: red → green. A test that has never failed proves nothing about your change — it may pass for any code at all. Run it against the pre-change code (stash the fix, or write the test first) and watch it fail for the right reason, then watch it pass. One red run is worth a hundred green ones.
7. A tool error means my change is wrong until proven otherwise. When the test runner, linter, or build errors, the reflex "the environment is broken" is how agents talk themselves past real failures. Investigate your own diff first; blame the environment only with evidence.
8. Pin before you refactor untested code (Feathers). If you're about to change code with no tests, first write a characterization test that locks in its current behavior. Then refactor. Otherwise "refactor" silently becomes "rewrite with regressions" and nobody notices until prod.

Red flags

• A test whose assertions all reference mocks/spies, never a real return value or state.
• One assertion per test that's just assert result is not None / expect(x).toBeDefined().
• Tests that re-state the implementation (same branches, same order) — they'll pass for a buggy rewrite that keeps the shape.
• 100% "coverage" with no edge cases — coverage measures lines executed, not behavior verified.
• A "done" / "production-ready" claim with no command output or run evidence anywhere.
• A new test that also passes on the unmodified code — it has never been red, so it doesn't test the change.
• A test/build failure waved off as "flaky" or "environment issue" with no investigation.
• Refactor of previously-untested code with no characterization test added first.
• Snapshot tests blindly regenerated on every change (rubber-stamping, not testing).

The Definition-of-Done evidence gate

A task is done only when you can paste evidence it ran — including the red run that proves the test is connected to the fix:

✅ Done. Evidence:
$ git stash && pytest tests/test_checkout.py -q       # before the change
... 1 failed (test_discount_applies) in 0.3s          # red, for the right reason
$ git stash pop && pytest tests/test_checkout.py -q   # after
... 14 passed in 0.4s                                  # green
$ curl -s localhost:8000/checkout -d '{"cart":...}' | jq .status
"confirmed"

No evidence → say "implemented but unverified," not "done." Honesty about an unverified state beats a false "production-ready." For schema/data changes, the evidence is a dry-run on a copy — see craft:data-and-state-evolution.

Inline example

// ❌ Tautological — passes even if applyDiscount returns garbage
it("applies discount", () => {
  const spy = vi.spyOn(pricing, "applyDiscount");
  checkout(cart, "SAVE10");
  expect(spy).toHaveBeenCalled();           // proves only that we called it
});

// ✅ Tests the behavior a user cares about
it("takes 10% off the cart total", () => {
  const result = checkout({ total: 200 }, "SAVE10");
  expect(result.total).toBe(180);           // fails if the math is wrong
});
it("rejects an unknown code", () => {
  expect(() => checkout({ total: 200 }, "NOPE")).toThrow(/unknown code/i);
});

Critique mode

For each test: what real bug makes it fail? If "none," flag it. Check for mock-only assertions, implementation-mirroring, missing edge cases, whether any new test was ever seen red, and — for the change as a whole — whether there's evidence it actually ran. Tag [trustworthy-tests · tautological-test|no-edge-cases|never-seen-red| unverified-done|no-characterization-test · SEV].

References

• Characterization tests / seams: ../../references/PRINCIPLES.md §D (Feathers)
• The false-confidence data: ../../references/EVIDENCE.md
• Rules 10 & 11 + Definition of Done: ../../RULES.md