Test and oracle builder

Test and Oracle Builder

Goal

Produce a complete, reproducible test suite that allows a grader (human or automated) to evaluate student submissions without ambiguity. The suite must cover nominal cases, edge cases, and error cases.

Inputs

Read all of the following before generating any output:

• mission.json — level, constraints, evaluation_mode
• statement.md — the exam or TP subject that students will receive
• rubric.md — grading criteria; tests must be traceable to rubric criteria
• reference/<solution> — the reference implementation produced by reference-code-engineer

Output contract

Produce the following artifacts, placed under tests/:

File	Description
tests/test-matrix.md	Human-readable case matrix: one row per test case with input, expected output, rubric criterion ID, and pass/fail oracle
tests/run_tests.sh (or .py)	Executable script that runs all cases against a student binary and reports results
tests/cases/	Directory with one subdirectory per test case: input/, expected_output/, README.md

test-matrix.md format

| ID | Description | Input | Expected output | Criterion | Notes |
|----|-------------|-------|-----------------|-----------|-------|
| TC-01 | Nominal: single child process | ... | exit 0, log line present | CR-FORK | ... |
| TC-02 | Edge: empty input | ... | exit 1, error on stderr | CR-ERRHANDLING | ... |

Rules

• Every rubric criterion must be covered by at least one test case.
• Distinguish oracle tests (deterministic expected output) from property tests (behavioural invariants).
• For C/POSIX assignments: test on Linux; document platform assumptions in tests/PLATFORM.md.
• For Python assignments: test with the Python version stated in mission.json → constraints.
• Do not rely on fragile output matching (e.g. exact whitespace). Use normalised comparison or regex where appropriate.
• Edge cases must include: empty input, malformed input, resource exhaustion signals (SIGPIPE, SIGTERM), and maximum-size input.
• The oracle script must exit 0 if all tests pass, non-zero otherwise, and emit a structured summary (tests/results.json).

C++ testing conventions

• Prefer ctest as the execution entrypoint for automated validation.
• Ensure test instructions work from repository root with:
  • cmake -S reference -B reference/build
  • cmake --build reference/build
  • ctest --test-dir reference/build --output-on-failure
• Include at least one negative/error-path test case in tests/test-matrix.md.
• Do not rely on locale-specific output formatting in assertions.

Java testing conventions

• At minimum, provide a deterministic compile + run validation path.
• Preferred local baseline commands:
  • javac -d reference/build reference/src/Main.java
  • java -cp reference/build Main <sample-input>
• Ensure tests and sample inputs are self-contained in the repository (no network, no external services).
• Include at least one nominal case and one error-path case in tests/test-matrix.md.

Rust testing conventions

• Provide a deterministic Cargo validation path:
  • cargo build --manifest-path reference/Cargo.toml
  • cargo test --manifest-path reference/Cargo.toml
• Keep tests deterministic and avoid timing-sensitive assertions.
• Include at least one nominal case and one explicit error-path case in tests/test-matrix.md.
• Prefer small, focused unit tests for parsing and core business logic.

Anti-patterns to avoid

• Tests that only verify the happy path — always test error handling.
• Tests that depend on external network or filesystem state not reproducible in a grading environment.
• Oracles that compare floating-point output without tolerance.
• A test matrix that maps multiple rubric criteria to the same test — keep tests atomic.