Agentic exam prep

Agentic Exam Prep

Overview

An assembly line for open-book exam packs: an iterative MCQ drill loop that tightens recall and reveals cheatsheet gaps, a print-pack architecture, course-material mining, and cram-pod/PDF tooling. Course-agnostic in mechanics; the depth axes and sources are re-designed per course (two worked applications: the COMP4021 and COMP4431 arc repos in this org). The point is to drill 7–15 correctly-chosen questions, not 561 — selection and signal quality are the whole game.

HARD RULES (safety + quality rails — read first)

• Canvas (canvas.ust.hk): GET only. NEVER POST/PUT/PATCH/DELETE. NEVER start a quiz attempt. No exceptions you initiate. This is an always-enforce, catastrophic-if-missed rule — a mutation creates a graded submission record that cannot be un-created. Quizzes (incl. "Pop Quiz" / "Lecture Video" types) count for grade. To get quiz content, you do NOT touch the quiz — mine from slides / transcripts / past papers / GitHub mirrors instead.
  • There exists a practice-quiz pool-harvest technique that mutates state (it POSTs). You never run it and never decide it's safe. You cannot reliably tell graded-from-practice via a read, so you do not judge it — only the operator does. It runs ONLY when the operator, this instance, explicitly tells you to AND has confirmed the quiz is ungraded practice. Absent that exact instruction: treat it as forbidden and don't surface the endpoints.
• Don't surface-clone the sample/past paper. Concept overlap is fine; reusing its code/scenario stems is not.
• Don't ship a one-shot autonomous trivia file as primary material. Fully-autonomous self-graded subagents are LOW-TRUST (the 4021 trivia agent missed 4 of 7 worth-attempting traps). Build high-yield material interactively or with mid-flight review; diff against an answer key before declaring complete.
• Don't teach to the test when patching the cheatsheet — patch general-truth claims, not the user's specific blind spot.

The MCQ drill loop (the core)

One active 25-Q set at a time; everything else is source pool or graduated flashcards.

Initial spawn (once per course): read materials → group into modules → diagnostic gap probe → hand-author Set 1 + Set 2 as style anchors (don't delegate until tone is set) → dispatch parallel creator subagents for the depth-axis source banks → run verifier subagents (second pair of eyes — real, not theatre: caught wrong-answer-letter bugs) → concat all_notes.md for one-file Ctrl-F.

Refresh cycle (each round) — triage the user's attempt by reasoning quality, not just correctness (read the Thinking: line):

Outcome	Action	Tag
Correct + confident reasoning (named the rule / used cheatsheet)	Graduate to flashcard bank	—
Correct via elimination / lucky guess / blank Thinking	Retest (same concept, new shape) — letter right, rule not internalized	🔁
Wrong / skipped / "don't know"	Revamp (fresh shape, explain misconception)	🔄
Misread (right reasoning, wrong letter)	Cheatsheet strategy patch + revamp	🔄
Ran out of time	Carryover (no revamp)	📦

Then: patch the cheatsheet where a miss revealed a gap (the miss is the completeness signal) → build next set = revamps (you author inline — you know the misconception) + carryovers + pulls from source banks to refill to 25 → drain: delete pulled Qs from their source files (sources shrink monotonically) → verifier pass on freshly-authored revamps → update lineage/drain tables.

Headline metric: 🔁/🔄 repeat-test passes — miss in round N, pass the revamp in N+1 = concept internalized.

Set 0.x extrapolation track (after ≥2 rounds): group misses by cognitive failure shape (e.g. "mutation ≠ completion", "Pythonic-coercion leak", "adjacent-concept decision-tree gap", "API getter/setter asymmetry") and predict where the SAME shape will bite on un-drilled concepts. A Set 0.x miss = "the pattern hasn't generalized" = diagnostic gold. Out-of-syllabus is OK in moderation if flagged (≥70% in-syllabus, ≤30% flagged stretch).

Print-pack architecture

• Tag-freeze: after round-1 prints, git tag print-pack-v1-<date>. Never edit packed files in place (the printed copy is now the index of record); new material lands as to_print_md_roundN/ addenda with their own build_pack_roundN.py.
• Two-stage PDF for hard print-shop margins (Chrome --print-to-pdf silently shrinks-to-fit on overflow): Stage 1 pandoc+Chrome headless renders raw (@page{margin:0}, no margin-boxes); Stage 2 pypdf scale(0.93)-wraps each page onto a fresh letter (the scale envelope = inviolable ~0.30in white margin) + reportlab imprints headers in the outer band.
• Per-module N-up lectures: 9-up landscape, 3×3, slides rotated 90° CW; one PDF per module, not one book. Decide layout by visual A/B (render, look) and quantify utilization% first; confirm slide aspect (16:9 vs 4:3) before scaling.
• Pull-out maneuver: when the cheatsheet bloats with audience-specific content, pull detail to a dedicated primer and leave a one-line breadcrumb stub at the original spot ("references must be kept, else it's dangerous to read the other sheets with missing content").

Source mining

• Canvas (read-only — see HARD RULES): snapshot via the browser's own session — fetch(url,{credentials:'include'}) from a Canvas-origin Edge tab (HttpOnly cookie never leaves the browser). Works as a student: course/modules/assignments/files/folders/pages-by-slug/lecture-video file-IDs.
• Course site (e.g. course.cse.ust.hk/<code>/) when materials aren't on Canvas; check GitHub mirrors first (martin-mok/HKUST-COMP*) before any paywall.
• Past-paper syllabus-drift table before drilling any past paper: <year> topic | status in current syllabus — flag REMOVED topics (a 2020 paper had ~25% mass in removed topics).
• NotebookLM cram pods: 3 topic-coherent notebooks ≤20 sources each (quality degrades past ~20); Pod 1 = exam forensics (≤7: past papers + worked keys, "walk every question, don't pad"); tone = "TA the night before". 1-up PDFs only (verify pdftotext); markdown ingests natively.

Subagent dispatch + trust calibration

Trust ranking (highest 4021 post-mortem finding): interactive-build > vendor-sheet-with-annotation > autonomous-mined-trivia. Dispatch that works: exact paths + exact output shape + word budget + "OK | ISSUE: <one-line>" format + hard tool-call cap (≤25). Doesn't work: answer-inference when curated answers exist, naive Read on >10 MB JSONL (Bash-grep first), "summarize" without a target schema. Author revamps yourself — you know the user's specific misconception; a subagent doesn't.

Course adaptation

Domain-agnostic (use as-is)	Course-specific (re-design)
25-Q active set + module quota; graduate→revamp→pull→drain cycle	The depth axes (4021: footguns/code-trace/confusion-pairs/depth-lookup/stack-translate; a DSP course needs formula-recall / algorithm-step-trace / diagram-interpretation instead)
Verifier second-pass; flashcard graduation format	Source-of-truth files + the mock/sample paper for no-clone
Cheatsheet-patch-on-miss; all_notes.md concat; 🔄/📦/➕ tags	Stack-translate set (keyed to user's daily stack)

When NOT to apply

• One-off exam not part of a series (tag-freeze/round-addenda is overkill).
• Closed-book exam (print-pack matters less; lean into drills + cram pods).
• User explicitly opts out of a step ("skip the diagnostic probe, time is short") — follow that.