Test design

Test Design

Overview

Answers "what input should a test feed, what should it assert, and what should it mock". Does NOT cover the red-green-refactor rhythm (that is TDD).

Core principle: tests protect a behavioral contract that users/callers can observe — not implementation details, and never a coverage number. A good test is a falsifiable spec of the contract; a bad test restates the code in another syntax.

REQUIRED COMPANION: for the write-test-first rhythm, see tdd. For module boundaries when mocks multiply, see solid. This skill only governs the design quality of cases and assertions.

When to Use

• Adding or modifying unit/integration tests for a piece of code
• Reviewing someone's tests to judge whether they are effective or formalistic
• Deciding "is this code worth testing?" or "unit vs integration?"
• About to write a fake test under coverage pressure, deadline crunch, or sunk cost

When NOT to use: just running tests / reading failures → test-runner. Just following the test-first rhythm → tdd.

The Iron Law (non-negotiable)

Break one line of production code and the test protecting it MUST turn red.
If it can't, the test is formalism — delete it and rewrite.

After writing each test, mentally (or actually) break the logic it guards and confirm it goes red. Violating the letter of the rule is violating the spirit — "my coverage is fine" and "I follow the spirit" do not count.

Workflow (when invoked)

Announce "Using test-design to …", then create one todo per step:

1. Decide if it's worth testing (§1): complexity in own logic → unit test; in dependency wiring → integration test. Not worth it → don't write.
2. Write a behavior list (§2): from the contract only — "when …, it should …", including things with no dedicated branch (e.g. "don't delete my legit content").
3. Pick representative points (§3): equivalence classes / boundaries / decision table / state transitions / dirty input — one representative each, no homogeneous duplicates.
4. Write assertions (§4): assert return value / observable state / side-effect boundary only; narrow and hard; hand-write expected.
5. Decide mocks (§5): mock only the outside world; count ≤ 2-3; prefer Fake/Recording.
6. Mutation check (Iron Law): break production code, confirm each test goes red.
7. Scan the rationalization table + red flags; if any hit, stop and rethink.

§1 Is it worth testing (ROI, not coverage)

Criterion: is the complexity in the code's own logic, or in the interaction between dependencies?

• Own logic (algorithms, pure computation, domain models/state machines, branch-heavy policy/validation/degradation) → unit test, highest ROI.
• Dependency wiring (CRUD, A.get() → convert() → B.save() glue) → integration test. The risk of such code lives exactly where mocks replace it (dependency returns null, times out, throws, transaction not committed); a unit test only proves "it runs when everything is fine" = worthless.

Don't unit-test glue; don't integration-test algorithms.

§2 Find cases from a behavior list

Write "when …, it should …" from the contract only. Never count ifs in the source. The list MUST include items with no dedicated branch that users care about most. If a case was "written while reading the source", delete and rewrite it — it protects the implementation, not the contract.

§3 Five techniques to pick representative points

1. Equivalence partitioning: one representative per behavior group (a second in the same group = duplicate).
2. Boundary values: for every threshold test boundary-1 / boundary / boundary+1 (>=4 must test "exactly 3" and "exactly 4" — that's the only way to catch > vs >=).
3. Decision table: enumerate combinations when output depends on multiple conditions; must include "several hit at once"; use pairwise when conditions are many.
4. State transition: legal path + illegal transition (use after close) + idempotency (close×2) + repetition.
5. Dirty/malicious input: empty, null, oversized, huge JSON, broken structure (missing closing tag, nesting), wrong type.

§4 Assertions (positive recipe)

Assert only these three observable targets: ① return value / thrown error (type + message + code) ② observable state change ③ side-effect boundary (whether a request was sent, where to, argument shape).

• Narrow and hard: several precise small assertions > one big toEqual(whole object).
• Hand-written expected: expected = input.replace(sameRegex) asserts "the regex equals itself" — always green.
• Don't touch implementation details: private call counts / internal branches, unless "count/order" is itself the contract (retries exactly 3 times, must not double-charge).

§5 Mock rules

Only replace slow/flaky/side-effecting/expensive external collaborators. Not for bypassing hard-to-test design.

Do Mock	Never Mock
Network/HTTP/3rd-party SDK/LLM, DB/FS/MQ, clock, RNG	The unit under test, pure functions, value objects, schemas, simple in-module collaborators

Fidelity order: Fake (in-memory) > Recording/contract replay > Spy > Mock framework. Assert only contract shape:

// Good: assert the outward contract
expect(fetchMock).toHaveBeenCalledWith(
  expect.objectContaining({ method: "POST", url: expect.stringContaining("/nodes") }),
)
// Bad: lock implementation detail, red on any refactor
expect(buildPayload).toHaveBeenCalledTimes(1)

§6 User perspective

Assert from the real consumer's view: end user sees results; upstream API/agent sees fields and error codes; ops sees whether logs/state can localize a problem. Test name = role + scenario + expectation (forbid case 1 / should work); enter through the public entry point, don't drill into private functions.

§7 Shape selection

Logic-dense → pyramid (many unit tests at the base); business/CRUD → diamond (many integration tests in the middle, unit tests only for utils and core algorithms). Integration tests catch what unit tests never can: transaction commit, dropped JSON fields, param validation, real dependency failures.

Rationalization Table (how you lie to yourself before writing formalism)

Excuse	Reality
"Just add a test to cover this branch"	Coverage isn't protection. Ask "what breaks for the user?"; no risk → don't test.
"Computing expected with a function is more accurate"	That tests "the impl equals itself" — always green. expected must be a hand-written literal.
"Mock a bit more for cleaner isolation"	Mocking a pile = split the code or go integration; you replaced the most bug-prone spot with a fake.
"Might as well test this glue/CRUD too"	Its risk is entirely in the mocked-out deps; use an integration test.
"Testing one boundary is enough"	Off-by-one is caught only at the boundary.
"Testing these few cases is more thorough"	Multiple cases in one equivalence class are homogeneous duplicates = formalism.
"Shipping soon, verifying the call is enough"	verify(called once) protects nothing. Short on time → write less, not fake.
"Snapshotting the whole object is easiest"	Any unrelated field change breaks it brittly, and hides which contract failed.
"This test is hard to write, mock around it"	Hard to test = hard to use — a design signal; fix the code.
"Looks professional / 80% is required"	100 impl-copying tests protect nothing. Quality, not quantity.

Red Flags — stop immediately

• Adding a test just to light up a line/branch
• expected computed with production logic instead of hand-written
• A "unit test" mocking 3+ dependencies
• Assertions are all verify(...) / toHaveBeenCalled, none on return value/state
• Test names like test1 / should work / case 2
• Testing getters/setters/DTO conversion/pure glue
• "Breaking the code probably wouldn't turn this test red"

Any hit → stop, return to: "what user-visible bad outcome does this test prevent?" Can't answer → don't write it.

Worked Example (techniques strung together)

Under test: strip(prompt) removes evidence dumps and keeps short instructions; a paragraph with ≥4 [n] citations is dropped.

Behavior list → cases (not branch-following):

// role+scenario+expectation; narrow/hard assertions; hand-written expected
test("subagent receiving a prompt with task_result has the dump stripped but instructions kept", () => {
  const out = new HeuristicDumpStripper().strip(
    "Summarize in three sentences.\n<task_result>…huge dump…</task_result>\nPrefer the latest conclusion.",
  )
  expect(out).not.toContain("<task_result>")        // dump gone
  expect(out).toContain("Summarize in three sentences")  // leading instruction kept
  expect(out).toContain("Prefer the latest conclusion")  // trailing instruction kept
})

// Boundary values: threshold >=4, test "exactly 3 kept / exactly 4 dropped"
test("paragraph with exactly 3 citations is kept", () => {
  expect(strip("conclusion [1][2][3] still body")).toContain("still body")
})
test("paragraph with exactly 4 citations is dropped", () => {
  expect(strip("noise [1][2][3][4] should be removed")).not.toContain("should be removed")
})

// Dirty input + fallback: non-empty when everything is stripped
test("falls back to truncated original (non-empty) when stripping empties it", () => {
  expect(strip("<task_result>only dump</task_result>").length).toBeGreaterThan(0)
})

Mutation check: change hits.length >= 4 to >= 5; "exactly 4 dropped" MUST go red. All green → the threshold isn't tested → rewrite.

Checklist (run once per test)

• Worth testing? complex logic → unit; glue → integration
• Cases come from a behavior list, not copied from source branches
• One representative per equivalence class; boundary±1 per threshold; "several hit at once" covered
• One case each for empty/oversized/broken/malicious input
• Assert return value/observable state/side-effect only; narrow and hard; expected hand-written
• Mock only external IO/time/RNG; the unit's logic is real; mock count ≤ 2-3
• Test name = role+scenario+expectation; enter via public API
• Mutation check passes: break production code and this test goes red