Aise test

Test Candidate Changes

Validation Integrity

• Test one focused hypothesis at a time.
• Use raw prompts, files, outputs, traces, screenshots, commands, and verifier results.
• Do not give a forward-test agent the expected answer or intended fix.
• Use train/replay evidence to design a change and keep holdout/regression evidence independent.
• A model judging its own prose is not enough when a deterministic verifier exists.
• Preserve failed attempts; do not tune repeatedly against the holdout set.

Workflow

1. Read the candidate with physisai review --root <project-root> <proposal-id>.
2. Derive checks from the candidate's source, risk, claimed behavior, and the current project's real verification tools.
3. Run only checks that can materially prove or disprove the proposal. Examples include DDL/source inspection and SQL dry runs; requirement traceability and acceptance cases; lint, types, tests, browser screenshots and accessibility; API contract, migration, security and performance checks; or stable reproduction, negative boundaries and flake checks. For verified_artifact, tool_output, or test_result, inspect the live source and record source-verification=pass; a non-empty source_ref is only a pointer, not proof that the CLI checked it.
4. Record each required result:

physisai test <proposal-id> --root <project-root> \
  --check regression=pass \
  --check <task-specific-check>=pass \
  --evidence "<commands, artifacts, and result summary>"

5. Re-run physisai review and report every failed or missing gate.

For retrieval behavior, test both surfaces after approval in an isolated fixture: the representative physisai context --query must include the intended memory without unrelated project data, and generated/index.md must link its memory ID to the correct non-empty detail file. Expired, superseded, or rolled-back content must appear in neither surface. Do not use a semantic scorer as the baseline; compare any future retriever against this deterministic Markdown path.

For engine governance changes, also run python scripts/evaluate_governance.py. Treat its scenarios as checks of lifecycle guarantees only; they do not measure whether the host model became smarter or produced better answers.

physisai test records check assertions and supporting evidence. It does not execute the named commands itself, so include the actual command, exit code, and artifact or output summary whenever available. Treat validated as “all recorded gates passed,” not as cryptographic proof that an independent evaluator reproduced the result.

Boundary

A low-risk explicit user preference may pass deterministic source, secret, and conflict checks without a benchmark. Medium/high-risk workflow, tool, code, or guardrail changes require their declared checks. Never bypass a failed source, secret, conflict, regression, or security gate, and never promote from this skill.