Validation and testing

Validation And Testing

Purpose

Tests are a budget, not a virtue — spend them where failure hurts most. This skill decides what to test first (the highest-risk surface), how to test it with the least machinery (monkeypatch and fakes over mocking libraries), and when a test is not worth writing at all. It exists so that a change ships with exactly the verification it needs and no more.

When to Use / When NOT to Use

Use when:

• You made a non-trivial change (a branch, loop, parser, money path, or security boundary) and need one runnable check.
• You are deciding what to test first in a codebase.
• You are adding the first test to an untested project.
• You are choosing how to isolate an external dependency (network, LLM, DB, filesystem).

Do NOT use (load the complementary skill instead):

• You are diagnosing a failing test or a bug → debugging-playbook.
• You are deciding whether a change is safe to apply/commit → change-control.
• You are auditing security posture (threat model, boundaries) → security-review-playbook (this skill only tells you to test the boundary, not how to define it).
• You are setting up CI to run tests automatically → build-and-release.
• The change is a trivial one-liner with no logic → write NO test (YAGNI applies to tests too).

Core Methodology

Ground rules (each self-justifying, valid in any repository): prefer stdlib and monkeypatch over mocking libraries — fewer dependencies, less brittle setup; tests must run offline with no live services, so they pass anywhere; if a written threat model exists, test the boundaries it names — if none exists, use the risk ordering below; verification is proportional — at least one check per non-trivial change, and security-boundary changes take the full rejection-path set (see security-review-playbook Step 5).

Risk-first ordering

Test in this order; stop when your test budget for the change is spent:

1. Security boundaries — signature/HMAC verification, auth, input validation, path sanitization, deserialization. A hole here is an incident.
2. Parsers and untrusted-input handlers — anything that turns bytes/strings/JSON from outside into internal objects. Test malformed, oversized, embedded, and nested inputs.
3. Money / irreversible-effect paths — payments, deletes, publishes, migrations. Wrong output is unrecoverable.
4. Core domain logic with branches — ranking, dispatch, state transitions, retry logic.
5. Everything else — usually not worth a dedicated test.

Runbook

1. Classify the change. Trivial one-liner (rename, constant, log line, pure passthrough) → no test; state that. Otherwise continue.
2. Locate the highest-risk surface the change touches. Use the risk ordering above. If the change touches a security boundary or parser, that is what you test — even if the change itself was elsewhere.
3. Write the smallest failing-if-broken check. One test that fails if the logic breaks. For a parser, that means the awkward inputs (empty, malformed, oversized, nested), not just the happy path.
4. Isolate dependencies with the least machinery (see decision tree). Prefer monkeypatch + tmp_path; hand-roll a fake object; reach for a mocking library only when neither works.
5. Keep it runnable offline. No live network, no real LLM, no real DB. Fake the boundary. Local-first means the test passes on a plane.
6. Run it and watch it fail first if you can. A test that has never been red proves nothing. Break the code, see red, restore, see green.
7. For untested codebases, bootstrap minimally (see below). One test file on the highest-risk surface, using the runner already implied by the ecosystem — do not add a framework.

Decision tree: how to isolate a dependency

Need to isolate something in a test?
├─ Is it a function/attribute you can replace?
│   └─ YES → monkeypatch it (pytest monkeypatch, or setattr).
│           Replace the LLM call, the network fetch, the clock.
├─ Is it the filesystem / a database file?
│   └─ YES → tmp_path (pytest) / tempfile. Real code, throwaway dir.
├─ Do you need an object with a few methods returning canned data?
│   └─ YES → hand-roll a tiny fake class. ~5 lines beats a mock DSL.
├─ Do you need to assert complex call sequences / spy on many calls?
│   └─ MAYBE → a mocking library is justified. First ask if the design
│             is too coupled to test simply (that's the real bug).
└─ Is it deterministic and cheap (pure function, in-memory)?
    └─ Don't isolate it. Call it for real.

When NOT to write a test — checklist

Skip the test if ALL of these hold:

• The change has no branch, loop, or parsing.
• It touches no security boundary, no money path, no irreversible effect.
• A type checker or the compiler would already catch a mistake.
• The blast radius of it being wrong is a cosmetic or trivially-reverted issue.

If any box is unchecked, write the one check.

Bootstrapping tests into an untested codebase — checklist

• Identify the single highest-risk surface (use risk ordering) — that is test #1, not "coverage".
• Use the runner the ecosystem already implies (pytest if pytest-shaped; node --test for Node with zero deps; check package.json/pyproject.toml before adding anything).
• Add NO mocking framework — monkeypatch + tmp_path / hand-rolled fakes.
• Make the first test cover the awkward inputs, not the happy path.
• Confirm it runs offline with the repo's documented test command.
• Do not chase a coverage number; chase the risk.

Discovery Commands

# --- Is there already a test suite, and what runs it? ---
ls tests/ test/ 2>/dev/null                   # common test directories
find . -name 'test_*.py' -o -name '*_test.py' -o -name '*.test.*' 2>/dev/null
grep -n '"test"' package.json                 # Node: what does `npm test` actually run?
grep -A3 '\[tool.pytest' pyproject.toml setup.cfg 2>/dev/null   # pytest config

# --- Find the highest-risk surfaces to test first ---
grep -rniE "hmac|compare_digest|verify|signature|auth|token" --include='*.py' .   # security boundaries
grep -rniE "json.loads|json.parse|parse|deserialize|pickle|np.load" .             # parsers / untrusted input
grep -rniE "delete|publish|upload|charge|payment|migrate|drop table" .            # irreversible / money paths

# --- Run tests (ecosystem variants) ---
pytest tests/                                 # Python, all tests
pytest tests/test_security.py -v              # one file, verbose
pytest -k "signature" -x                      # only matching tests, stop on first fail
npm test                                       # Node (may be jest / vitest / node --test)
node --test                                    # Node stdlib test runner, zero deps
pnpm test                                       # pnpm monorepos

# --- Confirm a test actually fails when the code is broken ---
# (temporarily break the code, run the single test, expect RED, then restore)
pytest path/to/test.py::test_name -x

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
90% coverage, an incident still shipped	Tested easy code for the number, skipped the boundary	Risk-first: security/parser/money before anything else
Test suite needs the network / a live LLM to pass	Didn't isolate the external boundary	monkeypatch the call; tests run offline
A brittle wall of mock setup nobody understands	Reached for a mocking library by reflex	monkeypatch + a 5-line fake class; mocks only for genuine call-sequence assertions
Parser test only checks valid input	Tested the happy path	Test empty, malformed, oversized, embedded, nested inputs
Green test that never caught anything	Never saw it fail	Break the code once, confirm red, restore
Trivial getter has a 30-line test	Wrote a test YAGNI forbids	No test for logic-free one-liners
Untested repo gets a giant test PR that stalls	Tried to cover everything at once	One test on the highest-risk surface; grow from there
Fix shipped with no check, regresses later	Skipped the runnable check	Every non-trivial change leaves at least one check behind

Repository Examples

Repo facts below are EXAMPLES of the methodology, not assumptions for other repos.

PRISM — the whole suite IS the highest-risk surface (as of 2026-07-04) ~/prism (Python 3.10+ FastAPI PR reviewer) has exactly ONE test file, tests/test_security.py, covering exactly the highest-risk surface: HMAC signature verification, repo-name validation, and payload size limit. Run with pytest tests/. This is risk-first ordering taken to its logical end — the security boundary is tested and little else is. The hardening commit 51b92c9 was locked in by test commit b011239 "tests: add security path coverage for signature, repo validation, size limit". Its patterns are testable-by-design: hmac.compare_digest before JSON parse, size check before json.loads, isinstance guards after json.loads. No CI — the local pytest run is the whole safety net.

AGENTIX — monkeypatch + tmp_path, zero mocking libraries (as of 2026-07-04) ~/agentix (~700-line ReAct agent framework) tests with pip install -e ".[dev]" then pytest. Its test pattern is the doctrine verbatim: pytest monkeypatch for the LLM and network, tmp_path for the DB, NO mocking libraries. Coverage maps to the risk surfaces:

• tests/test_agent.py — JSON extraction across plain/fenced/embedded/nested/malformed inputs (a parser — so the awkward inputs are tested), tool dispatch, unknown-tool handling, and malformed-JSON retry (1 strike recovers, 2 strikes surfaces the raw output).
• tests/test_tools.py — registry, shell blocklist, file read/write, the 50KB size cap, depth-2 directory listing, python exec, web-search fallback (a security/boundary surface).
• tests/test_memory.py — rolling window, cosine similarity, top-k ranking, clear.

RAGIT — the cost of no tests (as of 2026-07-04) ~/ragit (local RAG CLI) has NO tests and no CI — described as the owner's known biggest risk at v0.1.0. The numpy-2.0-breaks-chromadb-0.4 incident (a1ad63ed then 0a06ae9b) was caught by manual testing, not automation, and required a fix-after-fix. This is the concrete argument for bootstrapping: one test exercising indexer.py's embed-and-store path (it batches embeddings 64 at a time) would have caught the runtime break.

NYTW — the runner IS the ecosystem's, with zero deps (as of 2026-07-04) ~/NYTW runs its quiz tests via npm test from quiz/ using the Node native test runner — zero test-framework dependencies. This is "use the runner the ecosystem already implies, add no framework" in practice. Grading uses strict-JSON LLM output with enum validation ({"score":"correct|partial|incorrect"}) — a parser/boundary worth testing with fake LLM output.

Validation Criteria

You applied this skill correctly if:

1. You can name the risk tier (security / parser / money / domain / other) of the surface you tested and why it was the highest one the change touched.
2. Your test isolates external dependencies with monkeypatch/fakes/tmp_path, not a mocking library (or you can justify the library).
3. The test runs offline with the repo's documented command.
4. You saw the test fail before you saw it pass (or can explain why not).
5. A trivial one-liner change has NO test, and you said so explicitly.
6. For a parser, your test includes malformed/oversized/nested inputs, not just valid ones.
7. At least one runnable check accompanies each non-trivial change — never zero; security-boundary changes carry the full rejection-path set (see security-review-playbook), everything else stays proportional.

Provenance & Maintenance

Sources: ~/prism, ~/agentix, ~/ragit, ~/NYTW, investigated 2026-07-04. Methodology is repo-independent; the monkeypatch-over-mocks and risk-first doctrines are confirmed against agentix and prism.

Assumptions made:

• All repos are local-first with zero CI; the local test command is the entire safety net.
• ragit's "one test would have caught it" claim is illustrative reasoning, not a change that was made.
• Test file contents for agentix/prism reflect the fact pack as of 2026-07-04; not re-read in this session (file access unavailable at authoring time).

Re-verification commands:

ls ~/prism/tests                    # expect: test_security.py (only)
ls ~/agentix/tests                  # expect: test_agent.py test_memory.py test_tools.py
grep -rn "monkeypatch\|tmp_path" ~/agentix/tests   # confirm no mocking library
grep -n '"test"' ~/NYTW/quiz/package.json          # confirm node --test
ls ~/ragit/tests 2>/dev/null        # expect: nothing (no tests)

Likely to drift: ragit may gain tests (its owner knows it's the top risk); NYTW/ruflo test runners may change (ruflo uses Vitest); agentix test file contents evolve with the framework.

Maintenance checklist:

• Re-run re-verification commands; update file lists if tests were added/removed.
• If ragit gains a test suite, move it from "cost of no tests" to a positive example.
• Confirm agentix still uses monkeypatch/no-mock-library (core to this skill's doctrine).
• Re-stamp Repository Examples with the new verification date.