Researcher

Researcher — Measurement-Driven Optimization Loop

Run an optimization research campaign against a real codebase with a measurable metric. Think → Test → Reflect, autonomously, with .lab/ as the untracked source of truth.

When to activate

• A measurable optimization target exists (memory, RSS, CPU, latency, binary size, allocs/op, startup time).
• The user wants iterative, evidence-backed improvement rather than a one-shot rewrite.
• There is a way to build and run the code and read a number out.

Do not activate for: bug diagnosis (use hypothesis-driven), trivial one-line fixes, or work with no measurable signal.

Core discipline (non-negotiable)

1. Commit before running. Every real experiment is committed with experiment: <desc> before measurement. This is the safety net.
2. Measure after. Run all measure commands (primary + secondary), record raw values.
3. Keep / discard by the primary metric. Improved past the noise threshold → keep. Equal or worse → git reset --hard HEAD~1 and log discard.
4. Log every result. A row in .lab/results.tsv and an entry in .lab/log.md. No result goes unrecorded.
5. Protect .lab/. It is untracked and survives all git operations. Never git clean over it. Use targeted resets.

Safety hardening for critical infrastructure

When the codebase is production-critical, add two co-gates alongside the metric:

• Tests stay green. The project's test/lint/coverage commands are part of every experiment's run chain. An experiment that improves the metric but breaks tests is a discard, no exceptions. Record the failure as the discard reason.
• Behavior is preserved. The metric win is only valid if behavior is unchanged. The existing test suite is the guardrail; if coverage is enforced (e.g. 100% thresholds), that enforcement catches drift. State the behavior-preservation falsifier up front: "this experiment is wrong if test X fails or behavior Y changes."

State these gates in .lab/config.md under constraints so every iteration honors them.

Workflow

Phase 0 — Resume check

If .lab/ exists: read config.md, results.tsv, branches.md, tail of log.md. Summarize objective, metrics, best vs baseline, last status. Ask: resume or start fresh? On fresh: archive to .lab.bak.<ts>/.

If not: Phase 1.

Phase 1 — Discovery

Interview the user (skip what's obvious from context). Use defaults when the user has no preference:

1. Objective — one sentence.
2. Primary metric — required, drives keep/discard. Name, measure command (outputs a number), direction (lower/higher better).
3. Secondary metrics — tracked for context, only break ties.
4. Scope — files/areas modifiable.
5. Constraints — off-limits, plus the two safety gates above for critical code.
6. Run chain — the full command sequence to build + run one experiment and emit the metric(s). Entire chain must succeed.
7. Wall-clock budget per experiment — default 5 min; raise for long-settling runtimes (GC, scavenger, JVM).
8. Termination — target value, experiment count, or "plateau / diminishing returns / user interrupt."
9. Load validity — confirm the bench exercises the production load paths, not just idle polling. If the system serves traffic, the bench must drive that traffic (HTTP scrapes, requests, real work) at a realistic cadence; idle-only measurements miss the serving-path allocations and mislead. State explicitly which production paths the bench drives and which it doesn't.

Repeat the config back; get explicit confirmation before Phase 2.

Phase 2 — Lab setup

1. git checkout -b research/<slug> from current HEAD.
2. Create .lab/ at repo root.
3. .lab/config.md — all agreed parameters, baseline + best placeholders.
4. .lab/results.tsv — header: experiment\tbranch\tparent\tcommit\tprimary\tsecondary\tstatus\tduration_s\tdescription. Status: keep|discard|crash|thought|keep*|interesting.
5. .lab/log.md.
6. .lab/parking-lot.md.
7. .lab/branches.md — Branch, Forked from, Status, Experiments, Best metric, Notes.
8. Add .lab/ and run.log to .gitignore.
9. Production-build fidelity gate (do this before the baseline). Confirm the bench builds the binary the same way production does — same build flags, same static/dynamic linking (CGO_ENABLED), same GOOS/GOARCH, same strip level. If production builds via a CI workflow and the bench builds via a local go build, diff the flags. A baseline measured against a local cgo build while production ships a static binary will be inflated by libc RSS that production never has, and any "win" that's really just fixing that discrepancy is an artifact, not a production improvement. Record the confirmed build flags in .lab/config.md.
10. Run experiment #0 = baseline (no changes). Record it. Fill baseline + best in config. Sanity-check the baseline number against the production-build fidelity finding before trusting it.
11. Begin.

Phase 3 — Autonomous research

THINK — read results.tsv, last 5 log.md entries, branches.md, parking-lot.md, and in-scope source. Re-read discipline. Analyze, hypothesize, check convergence signals.

TEST — implement one logical change, commit, run the full measure chain, record raw values.

REFLECT — what confirmed, what surprised, what breaks the model. Update parking lot.

For every real experiment:

1. git commit -m "experiment: <desc>" before running.
2. Run the entire run chain (build + tests + measure). All must succeed.
3. Decide: keep / keep* (primary up, secondary regressed — log trade-off) / discard (reset --hard HEAD~1) / interesting / crash (reset, read last 50 lines of run.log; trivial→fix&rerun once, fundamental→move on, 3 crashes→rethink) / timeout (kill, log crash, reset; 2 in a row→reassess).
4. Append results.tsv row + log.md entry.

Log entry format:

## Experiment N — <title>
Branch / Type (thought|real) / Parent (#M) / Hypothesis / Changes / Result / Duration / Status / Insight

Autonomy

Default: work autonomously, log instead of reporting. Consult the user only when (a) the only viable path needs files outside scope, or (b) all strategies, branches, and parking-lot ideas are exhausted.

For critical infrastructure, soften this: check in more often, especially before declaring a floor or stop condition. A declared "floor" is a hypothesis, not a conclusion — invite the user to challenge it, and re-examine the space (binary composition via go tool nm/pmap, runtime chunks, every mapping) before concluding. Two expensive mistakes come from declaring the floor too early: missing a real lever, and shipping a "win" that's actually a bench artifact. The user's pushback is signal, not noise.

Branching

Fork when an approach diverges fundamentally or a branch stagnates. git checkout <keep-commit> → git checkout -b research/<new-slug> → register in branches.md. Consider all branches when thinking. Mark exhausted branches closed.

Re-validation

Every 10 real experiments: re-run current HEAD, compare to recorded best. If regressed >2%, log drift and consider forking from best.

Phase 4 — Wrap-up

On termination/interrupt: re-validate global best, write .lab/summary.md (total experiments, keeps/discards per branch, best vs baseline, top impactful changes, genealogy, key insights, failed approaches, remaining parking lot), checkout best branch/commit, report concisely.

Convergence signals

Signal	Action
5+ discards in a row	Approach exhausted; pivot
Metric plateau (<0.5% over 5 keeps)	Try something radically different
Same code area modified 3+ times	Explore elsewhere
Results contradict theory	Model is wrong — rethink
2+ timeouts in a row	Approach too expensive
Fighting the language/runtime to move the needle	Likely at the floor; consider stopping

The last signal is the "stop before rewriting in another language" guardrail. If the only remaining wins require abandoning the implementation language or core dependencies the user declared in-scope, pause and report: we've reached the practical floor for this stack. Don't drift into a rewrite unless the user explicitly green-lights it.

Hypothesis strategies (tools, not rails)

Ablation · Amplification (push what works) · Combination (merge wins across branches) · Inversion · Isolation (one variable) · Analogy · Simplification (remove complexity, preserve metric) · Scaling (order of magnitude) · Decomposition (split a big change) · Sweep (parameter across a range).

Output contract

When reporting: Metric (baseline → best, % delta), Changes (the kept experiments and what each did), Evidence (the measurements, including at least one experiment that ruled out a credible alternative), Limits (what wasn't tested, assumptions remaining), Next (smallest action to go further, or "at practical floor").

Pre-PR gates (run before filing each kept experiment)

1. Tests + lint green — non-negotiable for critical code; a metric win that breaks a test is a discard.
2. Verify it actually ships — for build-flag, config, or toolchain changes: trace the change to the production build path, not just the local Makefile. If production builds via a CI workflow, confirm the workflow uses the changed flags (or calls the Makefile that does). A Makefile-only change that production ignores is not a win — it's drift waiting to happen. This is the lesson from a campaign where a strip change landed in the Makefile but the release workflow kept shipping unstripped binaries.
3. Cross-model review — required when the change touches an output contract, swaps a dependency, or alters a behavior surface; optional for trivial mechanical changes (build flags, one-line refactors). Trigger rule: if a reviewer re-benching in a different merge order could see different behavior, or if the change replaces a library others depend on, get a second model's eyes on it. Skip it for changes where "it compiles and the tests pin the behavior" is sufficient proof.
4. Re-measure against the correct baseline — if the campaign discovered a production-fidelity gap after the baseline was set, re-cite deltas against a production-accurate baseline, not the inflated one. Don't ship a "−X%" that's really "−(X−1.2MB of libc that was never in production)."

Resources

• references/go-memory.md — Go-specific measurement: VmHWM/VmRSS/smaps_rollup, GODEBUG=gctrace=1, runtime/pprof heap profiles, GOMEMLIMIT/debug.SetMemoryLimit, binary stripping, pmap. Load when optimizing a Go binary.
• references/hypothesis-strategies.md — elaborated strategy playbook with examples.
• references/pr-template.md — copy-paste PR body for each kept experiment. Surfaces hypothesis, falsifier, measured baseline-vs-new table, reproduction commands, test/behavior-preservation evidence, security impact, limits. Use it verbatim per PR; append the target repo's own required PR fields (e.g. PCI Security Impact) when present.