CERES

CERES: Coursework Evidence Review & Evaluation System

Review untrusted coursework and project submissions with three separated roles and deterministic checks. Treat tests as evidence, not as the whole scoring standard, and keep environment failures distinct from student failures.

Non-negotiable rules

• Never execute a submission during the intake or rubric-analysis phase.
• Never expose teacher solutions, hidden tests, credentials, or another student's files to a grading agent or submission process.
• Never inject a real API key into student code. Mock external APIs for required behavior; use an approved disposable credential only for a separately authorized online check.
• Never use a student-produced score, trace summary, or report claim as runtime ground truth. In static scope, treat it only as submitted evidence, check internal consistency against frozen teacher data, and state explicitly that it was not reproduced. In offline or formal scope, replay or recompute it from frozen teacher assets before making runtime claims.
• Never award points without criterion-local evidence. Record uncertainty instead of inventing evidence.
• Never convert an environment/setup failure into a student deduction until a clean teacher-controlled reproduction shows the submission caused it.
• Never deduct points for a security, credential, API, network, or configuration finding unless the supplied rubric explicitly assigns points to that issue. Record it as a redacted diagnostic, remediation recommendation, and human-review signal when appropriate. If the same evidence causes a separate rubric-defined functional failure, score only that functional criterion and do not double-deduct the security finding.
• Never replace the three semantic review roles with a generated batch grading script, parser, keyword scorer, or OCR-only pipeline. Use scripts only for deterministic safety, execution control, arithmetic, state, and aggregation.
• Never treat extracted PDF text or OCR as a complete review when page layout, figures, images, screenshots, diagrams, or presentation slides may carry scoring evidence. Require the responsible subagent to render and visually inspect those artifacts.
• Never claim Microsoft Waza code compatibility or reuse unless the exact source, commit, license, and official command have been verified. Read references/waza-integration.md before configuring Waza.

Required inputs

Require these paths or explicitly mark them absent:

1. Student submission directory or ZIP.
2. Assignment specification and report requirements.
3. Trusted starter template and allowed-change globs.
4. Versioned rubric whose criterion points sum to assignment.max_points.
5. Public and hidden test commands, if any.
6. Runtime image or teacher-controlled environment.
7. Reference solution, if available. Keep it outside the submission runtime.
8. Evaluation scope: static for artifact-only review, offline for contract/plumbing evidence, or formal for a course-grade-authoritative run.

Use assets/grading-config.example.json as the base contract. Read references/input-contract.md when adapting it.

Workflow

1. Freeze inputs

• Copy teacher inputs into a read-only evaluation area.
• Hash the rubric, template, tests, submission archive, and report.
• Extract ZIP/RAR files only with scripts/safe_extract.py. For HW3, pass both configured exclusions (code&result/data/** and **/data/**) because some archives rename the wrapper directory; excluded members are metadata-only and are never extracted.
• Treat any archive extraction failure as not_assessed, never as an empty submission or a numeric zero. A missing/incompatible RAR backend is infrastructure_error; route it to human_review, retry in a verified RAR-capable environment, or request a ZIP resubmission. Do not start grading agents from a partial extraction directory.
• Treat a required-artifact filename mismatch as reviewable evidence, not an automatic zero. If intake has already read a clearly named extensionless candidate and a static parser confirms that it is the requested source type, record the selected path and alias reason, then grade its content normally. Never execute the candidate to identify it.
• Validate configuration before analysis:

python scripts/validate_inputs.py --config grading-config.json

Stop for teacher clarification when rubric arithmetic is inconsistent, allowed-change boundaries are missing, or required artifacts are undefined.

Set evaluation.scope and evaluation.course_grade_authoritative explicitly. Static and offline results may be final within their declared scopes, but must never be presented as authoritative course grades.

2. Run intake gates

Create a redacted manifest without executing code:

python scripts/inspect_submission.py \
  --config grading-config.json \
  --submission-root submission \
  --out intake-manifest.json

Hard-fail intake on path traversal, symlinks, archive bombs, exposed likely secrets, protected-file changes, or configured size limits. A hard fail routes to human review; it does not automatically imply a zero.

For scope=static, continue to evidence grading after a redacted intake hard fail, but never execute or import submitted code. The final decision remains human_review.

For a formal run, include every required report, trace, recording, and source artifact in assignment.required_artifacts. Missing required artifacts produce review, which prevents automatic finalization.

3. Run the three roles

Use separate, isolated subagent sessions for all three semantic roles. Run Agent 1 once per shared assignment version. For each submission, start a fresh Agent 2 subagent; only after its outputs validate, start a different Agent 3 subagent. The parent agent may coordinate state and deterministic commands, but must not perform or script the semantic grading itself. If the host cannot start the required subagents, stop before semantic scoring and report the limitation; do not silently fall back to a parent-only or script-only grade. Exchange only the artifacts defined in references/agent-contracts.md.

• Agent 1 - Assignment Analyst: read all assignment materials, including PDF pages, images, diagrams, template structure, report rules, and rubric. Produce assignment-contract.json and test-plan.json. Do not score or execute.
• Agent 2 - Evidence Grader: inspect all scoring-relevant submission artifacts, including source code, rendered PDF pages, embedded figures, images, screenshots, diagrams, experiment results, reports, and presentations. Request only approved reproductions, and produce rubric-scores.json plus evidence.jsonl. Do not alter rubric weights.
• Agent 3 - Independent Auditor: receive the frozen contract, gate manifests, run records, scores, evidence, and the raw artifacts needed to verify disputed visual or document claims. Recheck arithmetic, provenance, contradictions, unsupported points, and evidence omitted from PDF/image review. Produce audit.json and the final narrative. Do not rely on Agent 2's hidden reasoning.

Give each role raw artifacts, not the previous agent's conclusions beyond the required structured handoff.

For PDFs and images, use the host's document, PDF, image, or visual-inspection capability as appropriate. Text extraction may help locate relevant pages, but it never replaces rendering and inspecting the pages or images that support a score.

4. Reproduce safely

Read references/security-and-reproduction.md before any dynamic check. Prefer a pinned Docker image with no network, no capabilities, a read-only submission mount, process/memory/CPU limits, and writable temporary storage only.

Preview commands first, then execute only after the isolation contract is satisfied:

python scripts/run_checks.py --config grading-config.json \
  --submission-root submission --out run-manifest.json

python scripts/run_checks.py --config grading-config.json \
  --submission-root submission --out run-manifest.json --execute

Use the deterministic coordinator when running the standard sequence:

python scripts/grade_pipeline.py --phase prepare \
  --config grading-config.json --submission-root submission \
  --work-dir grading-work --execute

For artifact-only grading, set scope=static, use runtime.backend=none, and omit --execute. The coordinator writes a not_run manifest and never invokes run_checks.py:

python scripts/grade_pipeline.py --phase prepare \
  --config grading-config.json --submission-root submission \
  --work-dir grading-work

After Agents 2 and 3 produce rubric-scores.json and audit.json, rerun with --phase assemble, --scores, --audit, and --final-dir. The assemble phase reuses the frozen intake and run manifests so audited hashes do not change. The coordinator does not replace the three semantic agent roles; it freezes and records the deterministic stages around them.

The runner refuses untrusted host execution. Host mode is allowed only for a configuration explicitly marked trusted_fixture: true; it provides process limits, not network isolation.

For LiveCodeBench prompt-engineering submissions, generate deterministic artifact evidence without running the evaluator:

python scripts/inspect_livecodebench_results.py \
  --expected-question-count 45 \
  --outputs-dir submission/outputs \
  --out static-result-checks.json

This output-only mode derives a common question-ID set across methods and must state that it was not compared with template questions. Do not read an assignment-excluded questions/data file to strengthen the check.

Run tests by rubric slice where possible. A single early failure must not erase partial credit for independent criteria. For stochastic agent behavior, use deterministic mocks, fixed seeds, multiple trials when specified, and report the distribution rather than selecting the best run.

5. Score from evidence

For every criterion record:

• earned_points and max_points
• status: pass, partial, fail, not_assessed, or infrastructure_error
• the explicit rubric requirement, the observed submission behavior/artifact, and the concrete gap between them
• evidence references with file/line, test ID/log, trace step, or report page
• reproduction command ID and exit code when applicable
• structured deductions whose points sum exactly to max_points - earned_points; each deduction states the reason, evidence refs, and what would restore the points
• confidence and unresolved questions

For every individual or cohort CSV, emit two adjacent columns per criterion: <criterion_id>_score and <criterion_id>_comment. Put the earned and maximum points in the score column. Make the comment suitable for direct student feedback by stating the rubric requirement, observed code/report evidence, exact gap, and a concrete improvement. Name the relevant file, line, behavior, test, or report section when the evidence provides it. Do not replace the gap with a description of the current implementation or vague language such as "the report is weak" or "the code needs improvement."

Do not use pass for a non-full deductive criterion. Tiered criteria may award less than their maximum while passing the attained tier, but must be configured with scoring_mode=tiered. If a non-full criterion lacks an explicit requirement, gap, quantified deduction, or recovery condition, route the report to human_review instead of publishing it as final.

Prioritize evidence in this order: teacher-controlled hidden test, deterministic replay, focused unit/contract test, static source inspection, report evidence, student-authored screenshot or claim.

6. Audit and assemble

Run the deterministic arithmetic/evidence audit after Agent 3's review:

python scripts/assemble_report.py \
  --config grading-config.json \
  --scores rubric-scores.json \
  --intake intake-manifest.json \
  --runs run-manifest.json \
  --audit audit.json \
  --output-dir final

This writes final-report.json, a standalone evidence-bearing final-report.md, and grade-summary.csv. Do not hand-edit the machine-readable score after assembly; revise the source score or audit artifact and rerun.

For a cohort, aggregate completed reports:

python scripts/aggregate_reports.py \
  --reports submissions/*/final/final-report.json \
  --csv cohort-grades.csv --jsonl cohort-grades.jsonl

For isolated resumable batch grading, initialize the scheduler only after a real Luna trace supplies the effective model and reasoning effort:

python scripts/batch_grade.py discover \
  --submissions-dir submissions --work-dir batch-work \
  --effective-model gpt-5.6-luna \
  --effective-reasoning-effort low

For each job, write prepare/archive-manifest.json with safe_extract.py --out and prepare/intake-manifest.json with inspect_submission.py --out. status keeps the job at discovered until both manifests validate, then advances it to prepared. If extraction reports rejected or infrastructure_error, status advances only that job to preparation_failed; no grading role may claim it, and assemble emits a human_review item with earned_points=null and score_policy=no_automatic_zero. Write Agent 1 outputs once to batch-work/assignment/roles/agent1/. Use status, then have workers call claim --role agent2; write only to the returned role_dir, refresh long-running work with heartbeat using the returned lease_id, and call complete with that same ID. On a task-local failure, call release with the same role, job ID, and lease ID; a stale worker must never release a replacement worker's lease. Repeat for agent3, then place each deterministic final report inside its job directory and run batch_grade.py assemble. Never share writable role directories or claim Agent 3 before Agent 2 completes.

Keep the lease timeout longer than the heartbeat interval and the expected provider reconnect window. status --lease-timeout-seconds 0 is a destructive recovery operation: use it only after confirming the owning runner process group and all model children have stopped. A worker supervisor must replace task loops that stop after local validation failures, while invocation timeouts must terminate the entire per-task process group before a replacement is claimed.

The scheduler requires explicit effective profile values, records requested/effective model and reasoning separately, and refuses a strict mismatch. Read references/luna-cli-orchestration.md before launching custom Luna roles through a CLI-injected configuration. Do not describe Waza's mock parallel executor as a real grading agent run.

If frozen evidence or grading policy changes after a job has completed, requeue it through the scheduler instead of editing a final report or forcing a backward state transition:

python scripts/batch_grade.py requeue \
  --work-dir batch-work --job-id STUDENT_ID \
  --reason corrected_static_evidence

requeue accepts only an unleased assembled_ready or assembled job, archives the prior canonical role outputs and final report, and returns the job to agent2_ready. Existing attempt directories remain immutable audit evidence.

Gate decisions

Gate	Hard stop	Human-review trigger
Contract	Invalid rubric sum; missing assignment ID	Ambiguous criterion or report requirement
Intake	Traversal, symlink, likely secret, protected-file edit	Missing optional artifact; unusual binary
Execution	Isolation unavailable; forbidden network/credential need	Dependency or platform mismatch
Scoring	Out-of-range points; positive score without evidence	Low confidence or conflicting evidence
Audit	Total mismatch; unsupported score change	Material disagreement between agents

Hard stops block automatic finalization, not necessarily the student's eventual grade.

Resource routing

• Read references/input-contract.md to create or validate a grading configuration.
• Read references/agent-contracts.md before starting any of the three roles.
• Read references/security-and-reproduction.md before running student code.
• Read references/scoring-policy.md for partial credit, reports, stochastic systems, and batch consistency.
• Read references/sample-notes.md when grading Tool Calling, RAG, or ReAct 8-puzzle assignments like the bundled examples.
• Read references/waza-integration.md only when configuring or claiming Microsoft Waza integration.
• Read references/luna-cli-orchestration.md before using Luna custom subagents or interpreting model-profile evidence.
• Copy assets/report-template.md only when a human-readable report needs customization.

Validation

After changing this skill, run both the Codex structural validator and its deterministic tests:

python /path/to/skill-creator/scripts/quick_validate.py /path/to/ceres-review
python scripts/self_test.py
python scripts/validate_waza_evals.py --eval evals/eval.yaml