agentsclimarketplace

Auto research engineer

Skill tpop78/memory-os/skills/auto-research-engineer

Use to autonomously optimize ONE asset against a single objective metric — change it, score it with a locked measuring stick, keep the winner / revert the loser, log each round, and repeat overnight until a target/plateau/wall-clock stop. Adapted from Karpathy's autoresearch loop.From its SKILL.md

Install
npx -y skills add tpop78/memory-os --skill auto-research-engineer

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

4.9 KB, as published. Nobody here has run it

Auto Research Engineer

You optimize ONE asset toward ONE number. You change the asset, score it, keep what wins, revert what loses, log every round, and keep going — unattended — until a stop condition fires or the human stops you. The three-file system lives in .memory/autoloop/<tag>/.

The three files

  • INSTRUCTIONS.md — the goal, the metric (+direction +target), the asset path(s), the budget, the stop conditions, the rules. Human-locked: you never edit it.
  • SCORING.sh — the locked measuring stick. Run it; read the single number it prints (the last numeric token on its last line). You may read and run it; you must NEVER edit it. No moving the goalposts — the definition of "better" is fixed.
  • RESULTS.tsv — the ledger you append to. Columns: round ref score delta cost_s status change.

Setup (only if not already scaffolded — normally /auto-research did this)

Scaffold with node ${CLAUDE_PLUGIN_ROOT}/hooks/scripts/autoloop-init.mjs <tag>, then help the human fill INSTRUCTIONS.md and implement SCORING.sh. Run the FIT CHECK below before looping.

FIT CHECK (run before the first loop; be honest)

A target is only worth optimizing if all three MUST-HAVES hold:

  1. Objectively scored — a real number, not "make it look nicer".
  2. Fast feedback — minutes/hours, not weeks (no SEO-reindex / multi-month churn).
  3. Write access — you can actually change the asset (a file/API, not a published video). NICE-TO-HAVES (more = more powerful): high feedback volume; cheap to fail; a consistent, repeatable measuring stick. If a must-have fails, say so plainly and propose a better-shaped target instead of pretending.

Mode (set in INSTRUCTIONS)

  • git-worktree — create a dedicated autoresearch/<tag> worktree after the human confirms the run. Snapshot only the declared asset paths before each round; commit only those paths for a winner and restore only those paths from best/ for a loser. Never run repository-wide reset or clean.
  • snapshot — not in git: before each change, the current best lives in .memory/autoloop/<tag>/best/; save each round to rounds/NNN/, promote a winner to best/, restore best/ to revert a loser.

The loop — repeat until a stop condition fires

  1. Integrity check. Re-hash INSTRUCTIONS.md + SCORING.sh; confirm YOU have not changed them. If the human changed them mid-run, reload and announce it.
  2. Baseline (round 0). Score the asset as-is (no change). Record it as the current best.
  3. One hypothesis, one change. Edit ONLY the declared asset path(s). Never touch any other file.
  4. Score. Run SCORING.sh within the per-round budget; read the single number. If it exceeds 2× the budget, kill it and treat the round as a crash.
  5. Decide (direction from INSTRUCTIONS):
    • better than best → keep: promote the round to best/, then commit only declared assets in git-worktree mode. New best.
    • not better → revert: restore only declared asset paths from best/.
    • crashed / non-numeric → if trivially fixable (typo, missing import) fix and retry once; else crash: revert and move on.
  6. Log one row to RESULTS.tsv (status = baseline|keep|revert|crash). Append a one-line note to .memory/JOURNAL.md and update .memory/STATE.md (Now = "auto-research <tag>, best=<score>").
  7. Check stop conditions from INSTRUCTIONS: target_score reached, plateau_rounds consecutive non-improvements, or wallclock_cap elapsed. If any fired → STOP and summarize. Otherwise loop.

Autonomy & guardrails

  • Do not ask "should I continue?" Once looping, run unattended until a stop condition or the human interrupts. This intentionally overrides memory-os's break_at checkpoint nudge for the run.
  • Write only the declared asset path(s). Never edit INSTRUCTIONS.md or SCORING.sh.
  • Stay isolated. Git runs happen only in the recorded dedicated worktree. Refuse to start if the worktree cannot be established or any declared path escapes it.
  • Preserve unrelated work. Never use repository-wide reset, clean, restore, checkout, or broad staging. Snapshot and restore only the declared assets.
  • Every loser reverts to a known-good state — never leave the asset broken.
  • Bounded: the stop conditions guarantee the run ends; never loop truly forever.

On finishing

Summarize: rounds run, baseline → best, total improvement, and where the winning asset lives (dedicated worktree branch HEAD or best/). Offer to write a short report and ask before removing the worktree.

Keep looking

Skills are one crate of 608,666. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.