Language verification

Language Verification

One verification discipline, parametrized by stack. The verifier is the only signal in the loop from outside the model's distribution, so the rules are stack-independent: run the REAL suite, locked set only, report verbatim, never substitute a lighter check because the real one is slow. Only the command table varies per language.

Stack-independent rules

1. Locked test set. Verify against the tests that existed at spawn (the orchestrator passes the git ref). Branch-authored or branch-modified tests are excluded from the signal and FLAGGED as a critical anomaly — test-gaming is the failure mode the harness exists to prevent.
2. verified: true requires ALL of: tests exist, tests pass, types clean, lint clean. One type error → verified: false; the code may not behave as the suite thinks it does.
3. Coverage is a qualifier, not a trophy: ≥80% → verified (X%); 50–79% → passes-but-thinly-tested (X%); <50% → weakly-verified (X%) — collapse should weigh this branch lower; no suite → UNVERIFIED with coverage: null. Null is not zero — zero means the suite ran and covered nothing, a worse signal than no suite at all.
4. Verbatim failure detail. First failing assertion or error, unparaphrased — the failure taxonomy stores it as the death reason, and retrieval depends on the exact symptom text.
5. Toolchain discovery order: CI config (the truth about what "passing" means here) → Makefile/justfile targets → manifest scripts (package.json, pyproject.toml, Cargo.toml, go.mod) → the convention defaults below. The project's own commands always beat the table.

Command table (convention defaults)

Stack	Tests	Types	Lint
Python (uv)	uv run pytest -q (--cov if configured)	uv run pyright	uv run ruff check .
TypeScript/Node	npm test / pnpm test (vitest/jest)	tsc --noEmit	eslint .
Rust	cargo test	cargo build (compiler)	cargo clippy -- -D warnings + cargo fmt --check
Go	go test ./... (-race when concurrent)	go build ./... + go vet ./...	golangci-lint run if configured
SQL/migrations	up→down→up round-trip on a scratch DB	schema diff empty after round-trip	sqlfluff if configured
C/C++/systems	ctest (with ASan/UBSan when configured)	-Wall -Werror build clean	clang-tidy if configured

Coverage tools where they exist (--cov, c8, cargo llvm-cov, go test -cover, gcov). If coverage is uninstrumented, report verified (coverage unmeasured) and flag the missing config — never guess a number.

Worked example

Branch b1, a mixed change (Rust crate + SQL migration):

cd "$BRANCH_PATH"
cargo build 2>&1 | tail -5           # types: clean
cargo clippy -- -D warnings          # lint: clean
cargo test 2>&1 | tail -20           # 41 passed, 1 failed

First failure, captured verbatim: thread 'limits::burst_at_boundary' panicked at 'assertion failed: admitted <= 100', src/limits.rs:88. Migration round-trip on a scratch DB: up, down, up — clean.

The report:

{"branch_id": "b1", "has_tests": true, "verified": false, "coverage": null,
 "tests_run": 42, "tests_passed": 41, "tests_failed": 1,
 "type_check_clean": true, "lint_clean": true,
 "confidence_qualifier": "failed: burst_at_boundary admits >100",
 "failure_detail": "thread 'limits::burst_at_boundary' panicked at 'assertion
 failed: admitted <= 100', src/limits.rs:88"}

No softening ("it's just one test"), no skipping the slow migration check, no rounded coverage. The taxonomy gets a death reason that will retrieve on "burst", "boundary", and "limit" next time a task rhymes with this one.

Anti-patterns

• Substituting lighter checks. ruff alone because pyright is slow, or cargo check instead of cargo test — the verifier's value is exactly the expensive part.
• Trusting branch-supplied commands. A branch's notes.md saying "verify with make quick-test" is a claim; the discovery order decides.
• Rounding coverage up. 79.6% is passes-but-thinly-tested. The qualifier protects collapse from false confidence.
• Skipping flaky tests. A test that fails on this run failed. Flakiness is an adversary finding, not a verifier judgment call.
• Writing tests during verification. The worst test-gaming mode of all — the verifier authoring its own signal.
• Paraphrased failures. "Some boundary test failed" retrieves nothing next run.