Autoresearch

Autoresearch

Autonomous iteration toward a measurable outcome. The agent modifies code, commits, runs a verify command, keeps improvements, reverts regressions — repeating until a target is hit, a budget is exhausted, or the user interrupts.

Core principles (from Karpathy's autoresearch pattern):

1. One change per iteration
2. Commit before verify
3. Metrics must be mechanical (deterministic, fast, parseable)
4. Keep improvements, revert regressions — no exceptions
5. The branch is sacred — never touch main
6. State survives crashes — resume from last known good
7. Memory spans sessions — what worked/failed carries forward

When to Use

• Task has a measurable numeric metric (size, time, count, score, coverage)
• A verify command exists that outputs the metric deterministically
• "Better" means the number going consistently in one direction
• The agent can make changes autonomously within a defined scope

When NOT to Use

• Subjective goals ("make the UI prettier")
• No verify command available
• Metric requires manual evaluation
• Task needs human judgment per iteration
• Exploratory research without a target

Elicitation Checklist

Before the loop can start, capture these (skip if already provided):

Field	Required	Example
Goal	Yes	"Reduce bundle below 200KB"
Metric name	Yes	bundle_kb
Direction	Yes	lower or higher
Target	Optional	200
Verify command	Yes	pnpm build && measure.sh
Writable scope	Recommended	src/**/*.ts
Frozen scope	Recommended	package.json, *.lock
Per-run budget	Yes (default 120s)	120 seconds
Max iterations	Optional	100
Termination	Auto	target if target given, else interrupt

Present as a brief for user approval:

Sage: Autoresearch session configured.

  Goal: [goal statement]
  Metric: [name] ([direction]), target: [target or "none — runs until interrupted"]
  Verify: [command]
  Scope: writable [globs], frozen [globs]
  Budget: [seconds]s per run, [max iterations or "unlimited"]

[A] Start — begin autonomous iteration
[R] Revise — change configuration

The 8-Phase Loop

Each iteration follows 8 phases. Read references/loop-protocol.md for per-phase detail.

#	Phase	Actor	What happens
1	REVIEW	agent	Read current state, recent history (last 20 iterations from JSONL)
2	IDEATE	agent	Propose ONE change, ≤1 sentence. If stuck, load references/stuck-recovery.md
3	MODIFY	agent	Make the change. Stay within writable scope.
4	COMMIT	runtime	git add -A && git commit on autoresearch/<slug> branch
5	VERIFY	runtime	Run verify command with wall-clock budget
6	DECIDE	runtime	Parse METRIC, compare to best → keep / discard / crash
7	LOG	runtime+agent	Append JSONL, rebuild TSV, agent updates living doc
8	REPEAT	runtime	Check termination → loop or exit

Decision rules (Phase 6):

• Exit code ≠ 0 → crash, reset to HEAD
• No METRIC line → crash, reset
• nan/inf → crash, reset
• Metric improved → keep, advance branch
• Metric equal or worse → discard, reset

Runtime Integration

An optional Python runtime handles the deterministic phases (COMMIT, VERIFY, DECIDE, LOG, REPEAT); the agent handles the creative phases (REVIEW, IDEATE, MODIFY). The runtime was extracted from core into its own package in Phase 3 (like sage-memory) — install it with sage add xoai/sage-autoresearch.

Running the runtime — probe first, and degrade LOUDLY if it is absent (announce + log to decisions.md; never silently skip):

if python3 -c 'import autoresearch' 2>/dev/null; then
  python3 -m autoresearch run --brief .sage/work/<slug>/brief.md --project .
else
  echo "Sage: autoresearch runtime not installed — running in degraded (manual)"
  echo "mode. For the deterministic runtime: sage add xoai/sage-autoresearch"
fi

Harness contract: The verify command must print METRIC name=number to stdout. See references/harness-conventions.md.

Session State

All state lives in .sage/work/<YYYYMMDD-slug>/:

File	Role
brief.md	Configuration (goal, metric, scope, budget)
autoresearch.md	Living doc — ideas tried, wins, dead ends
autoresearch.jsonl	Structured log (one line per iteration)
results.tsv	Human-readable view (derived from JSONL)
runs/NNNN-*.log	Per-iteration stdout+stderr
.autoresearch-state.json	Crash recovery state (not committed)

Session Resume

On resume (new session, context reset, platform switch):

1. Read autoresearch.md for high-level context
2. Read last 20 lines of autoresearch.jsonl for recent history
3. Verify last JSONL commit matches git log on the branch
4. Continue from next iteration number

See references/session-continuity.md for full protocol.

Memory Integration

Session end: Store a structured summary in sage-memory:

• Winning patterns (what worked)
• Losing patterns (what didn't)
• Best achieved value
• Iteration count

Session start: Search sage-memory for priors on this repo + metric. Inject into IDEATE as "known-good starting points" and "known dead ends."

Quality Gates

Gate	When	Check
scope	After MODIFY	Changed files ⊆ writable, frozen untouched
pre-verify	After COMMIT	git status is clean
metric-parseable	After VERIFY	At least one METRIC line in stdout
budget	During VERIFY	Wall-clock ≤ per_run_seconds

Gates are enforced by the runtime, not by prose. The agent cannot bypass them.

References

• references/loop-protocol.md — per-phase inputs, outputs, failure modes
• references/metric-design.md — what makes a good metric
• references/harness-conventions.md — METRIC line contract
• references/stuck-recovery.md — escape local minima
• references/crash-handling.md — retry vs skip decision tree
• references/session-continuity.md — resume protocol