Ai agents research frontier

AI Agents Research Frontier

<!-- vendor-portability: contributor-facing knowledge pack for the rjmurillo/ai-agents repo itself; intentionally references upstream paths (.agents/, .claude/, scripts/, build/) because its audience is repo contributors, not plugin consumers (issue #2050) -->

Open problems where this repository can advance the state of the art, ranked in owner-confirmed priority order. This skill tells you WHAT is worth working on and what "done" would look like. It does not teach experiment discipline (that is ai-agents-research-methodology) and it is not the portability battle plan (that is ai-agents-portability-campaign).

Honesty contract for this document: every asset claim below was re-verified against the working tree on 2026-07-30. Anything labeled PROPOSED is not policy. Anything labeled UNVERIFIED could not be confirmed and must be checked before you build on it.

Triggers

• research frontier
• open problems
• what should we research next
• frontier programs

The Three Programs at a Glance

Rank	Program	Anchor artifacts	Status (as of 2026-07-30)	Falsifiable milestone (short form)
1	Verified governance	ADR-069, scripts/eval/eval-rule-activation.py	ADR-069 is PROPOSED; eval tool exists, 15 rule scenario fixtures exist	Controlled eval shows gated-corpus sessions beat ungated on N scenarios with defensible stats
2	Cross-harness abstraction	ADR-072, ADR-068, build/generate_agents.py, tests/build_scripts/test_generate_hooks_runtime_contract.py	ADR-072 PROPOSED, ADR-068 ACCEPTED as of 2026-07-30; generators and contract tests exist and run	New harness target added with zero hand-edits to generated trees, contract suite green
3	Self-improving loop	issue #1345 hooks, EVENT= telemetry (retired, ADR-084/#5154)	Apply-step hooks unregistered; end-to-end consumer pipeline UNVERIFIED	A correction observed once auto-proposes a guard that survives calibration

Process

Phase 1: Verify current state before claiming anything

The evidence below decays. Re-run these before quoting any of it in an ADR, issue, or write-up. All commands run from the repo root.

Claim	Re-verify command
ADR-069 status is proposed	head -5 .agents/architecture/ADR-069-context-corpus-is-the-product.md
ADR-072 status is Proposed with approval conditions	sed -n '1,15p' .agents/architecture/ADR-072-jtbd-plugin-architecture.md
ADR-068 status is Accepted as of 2026-07-30	sed -n '1,10p' .agents/architecture/ADR-068-consolidated-hook-dispatcher.md
Rule-activation eval exists with a no-spend path	python3 scripts/eval/eval-rule-activation.py --help
Rule scenario fixtures	set -- tests/evals/rule-scenarios/*; echo $#
Corpus size across skills, rules, retros, memories	python3 -c "from pathlib import Path as P; print(len(list(P('.claude/skills').glob('*/SKILL.md'))),'skills',len(list(P('.claude/rules').glob('*.md'))),'rules',sum(1 for p in P('.agents/retrospective').glob('*.md') if p.is_file() and p.name != 'INDEX.md'),'retros',sum(1 for p in P('.serena/memories').rglob('*.md') if p.is_file()),'memories')"
Runtime contract tests pass	uv run pytest tests/build_scripts/test_generate_hooks_runtime_contract.py -q
Apply-step hooks unregistered	uv run pytest -q "tests/build_scripts/test_copilot_dispatcher_artifact.py::TestDispatcherArtifacts::test_retired_hooks_are_absent_and_keepers_are_plugin_only"

Phase 2: Pick a program

Take the highest-ranked program whose first unclaimed step you can finish. Rank order is owner-confirmed; do not reorder it without the owner. If your idea does not fit any program, it is a new capability: route through the buy-vs-build-framework Quick tier gate first (AGENTS.md Skill-First section), then ai-agents-research-methodology for the idea lifecycle.

Phase 3: Execute the first steps

Each program section below lists three concrete first steps IN THIS REPO. Steps are ordered; do not skip to step 3 because it is more interesting. Every step that produces a measurement must predict the number before running (see ai-agents-research-methodology), and every probe of external tool behavior must follow ai-agents-empirical-probe-toolkit (pinned version, foreign cwd/env, negative control).

Phase 4: Report and gate

Results go through the normal machinery, not around it: write-ups per ai-agents-docs-of-record, changes through ai-agents-change-control, ADR edits fire the adr-review debate gate. A negative result is a result: record it (the #2230 rejected-fix pattern) so the next person does not re-run it.

Program 1: Verified Governance (ADR-069, PROPOSED)

Thesis (quoting the ADR title verbatim): "The Curated Context Corpus IS the Product, Orchestration Is Plumbing" (.agents/architecture/ADR-069-context-corpus-is-the-product.md, status: proposed, date 2026-05-02). Core frame: no learning between runs; the only thing that persists is what re-enters the next context window, so the corpus is the durable competitive surface and everything else is plumbing.

Why current state of the art fails

• Prompt engineering is folklore: rules ship because they sound right, and almost nobody measures whether a rule changes model behavior at all.
• Rules are unmeasured even here: this repo carries more rule files under .claude/rules/ than scenario fixtures under tests/evals/rule-scenarios/. Run the Phase 1 corpus command for current numbers rather than quoting a stored one. Most rules have never had an activation baseline.
• Weight tuning is unavailable to a repository: you cannot fine-tune the vendor model, so context curation is the only lever, and the field has no shared methodology for verifying it.

This repo's asset

• A large corpus of skill directories, rules, retrospectives, and Serena memories. Run the Phase 1 corpus command for current counts.
• Gates that produce inspectable artifacts (verification-based governance): every rule violation leaves evidence, so compliance is measurable after the fact.
• A working eval harness: scripts/eval/eval-rule-activation.py compares baseline (no rule) vs description-only vs full-body loading, LLM-judged on activation, citation, and behavior scores (1 to 5), with --dry-run as the zero-spend validation path and ADR-035 exit codes.
• A measured failure baseline to beat: FM-1 (Context Reading Failure) recorded 95.8% session-start non-compliance in a sampled period (.agents/governance/FAILURE-MODES.md:44).

First three steps

1. Wire rule-activation baselines for 3 high-traffic rules that lack fixtures. UNVERIFIED: no per-rule traffic measurement exists; as a proxy, pick rules cited most often in .agents/retrospective/ (grep the rule filename). Write scenario JSON files modeled on tests/evals/rule-scenarios/refactoring.json, dry-run first, predict scores before the paid run.
2. Measure FM-1 compliance rate before and after one deliberate context change (for example, moving one rule between description-only and full-body loading), using session logs under .agents/sessions/ as the compliance record. Predict the delta first.
3. Publish the methodology doc: how this repo turns a governance rule into a measured, gated, evidenced control. Route it through adr-generator or a governance doc per ai-agents-docs-of-record; it is the publishable unit.

Falsifiable milestone

You have a result when a controlled eval shows a gated-corpus session outperforms an ungated session on N pre-registered scenarios with statistically defensible evidence (pre-declared N, pre-declared success metric, variance controls per scripts/eval/variance-control.py). You have a negative result, equally worth publishing, if the gated corpus shows no measurable difference: that would falsify the strong form of ADR-069.

Program 2: Cross-Harness Abstraction (ADR-072 seam, ADR-068)

Why current state of the art fails

• Every harness reinvents plugin semantics: Claude Code and Copilot CLI diverge on matchers, hook cwd, payload field casing, env anchors, and kill budgets (settled contract rows live in agent-harness-reference).
• No conformance suites exist: vendors ship docs that were wrong by omission at least twice for this repo (#2205 plugin-root env vars, #2290 payload casing); nothing in the ecosystem lets you certify "this plugin behaves identically on harness X and Y."

This repo's asset

• A working multi-target generator: build/generate_agents.py emits agent trees from templates/agents/*.shared.md, and build/scripts/build_all.py (7 generators) mirrors .claude/ content into src/copilot-cli/ and .github/instructions/, with --check as the CI drift gate. Existing target trees: src/vs-code-agents/, src/copilot-cli/, src/claude/ (manual, ADR-036).
• Empirically settled contract tests: tests/build_scripts/test_generate_hooks_runtime_contract.py pins the hook anchoring contract under foreign cwd/env with a negative control, plus scripts/validation/validate_hook_anchoring.py.
• A settled env-anchor decision memory: .serena/memories/decision-copilot-cli-hook-plugin-root-contract.md.
• ADR-068 has real measurements to build on: issue #2295 recorded 3 of 197 preToolUse invocations killed by the 2 to 3 second budget, and ~246 ms measured Python cold start per shim on Windows with 40 PreToolUse shims.

Label honestly: ADR-072 is PROPOSED with an explicit "APPROVE WITH CHANGES" review and five conditions to reach Accepted; ADR-068 reached ACCEPTED on 2026-07-19. No code moves on ADR-072 alone. Nothing in this program may route around ai-agents-change-control.

First three steps

1. Formalize the contract table into a versioned spec: extract the empirically-verified rows from agent-harness-reference and the decision memory into one spec file with a version field and per-row evidence links (probe retro, memory, or code path). Docs-say rows stay marked docs-say.
2. Build a conformance test runner per harness: parameterize the existing runtime-contract tests so the same suite executes against each target tree, reporting per-harness pass/fail instead of Claude-only.
3. Third-target dry run: pick one harness this repo does not yet emit for (ADR-072 names Codex CLI and Cursor as distribution harnesses), run the generator pipeline in --what-if/--check mode against a prototype emission profile, and log every place a hand-edit would have been needed.

Falsifiable milestone

A new harness target is added with zero hand-edits to generated trees and the conformance suite green on every harness. The milestone fails, and the abstraction is falsified, if the third target requires manual patching of generated output or a contract row that cannot be expressed in the spec.

Program 3: Self-Improving Loop (issue #1345)

Why current state of the art fails

• Agent memories decay: observations pile up but nothing promotes them, so the same correction gets re-learned every few sessions.
• Corrections do not become enforcement: the industry pattern stops at "write it to memory," which is advisory. This repo's own history shows advisory rules fail (FM-1 at 95.8% non-compliance) while gates hold.

This repo's asset

• The Detect-Log-Graduate loop is live: Detect via the reflect skill, Log via Serena observation memories, and Graduate via the skillbook agent. The advisory correction-applier and topical-memory-injection hooks were deleted (issue #3184) after being deregistered from both Claude source manifests. Retrieve corrections and topical memories explicitly through the memory or memory-search skill. The registration-state test named in Phase 1 guards their absence from both Claude source manifests and the generated Copilot manifest.
• Guard telemetry existed at the emitter: push_guard_base.py wrote a machine-parseable EVENT={...} line to stderr on every block and fail-open. That emitter and every guard built on it were deleted under ADR-084 (issue #5154). The skill that consumed the telemetry and its two build scripts (an aggregator and a classifier, both deleted with it) classified guards as Budding, Growing, Mature, Proficient, Inert, or Harmful from age, intercept count, and fitness, with explicit prune/promote actions. It was retired in the same change (issue #5154) once its only producer was gone: neither the emitter nor the classifier exists in this repo today.
• A reflexion write path: memory-reflexion (ADR-063) extracts episodes. Its derived causal graph was removed by ADR-089 for having no reader.

What is NOT yet built (state this in any pitch):

• UNVERIFIED: no automated EVENT= consumer pipeline was found under scripts/ on 2026-07-03; push_guard_base.py (deleted under ADR-084, issue #5154) described a pipeline that "greps for ^EVENT=" but the tier-classifier report appeared to be the only consumer, invoked manually. Both the emitter and that classifier are gone now, so there is no producer and no consumer.
• UNVERIFIED: Forgetful memory population size and freshness are unknown; do not assume Tier 1 supplementary recall works until you query it.
• The Graduate step has no calibration gate: nothing today forces a promoted pattern to prove it would have fired correctly on real history (#1989 M4 shipped a threshold that could never fire; #1887 Phase-6 audit found its own guards would have prevented 0 of 35 fix commits).

First three steps

1. Rebuild the telemetry loop from scratch: both the EVENT= emitter and its tier-classifier consumer are gone (ADR-084, issue #5154). A revival needs a new emitter on whatever guard exists at the time, a scheduled aggregator, and a record of where the events persist; there is no existing pipeline to "close", only the retired design to reference.
2. Add a calibration gate to Graduate: before any correction-derived guard ships, replay it against roughly the last 5 real PRs and show it fires where it should and stays silent where it should (recipe in ai-agents-empirical-probe-toolkit).
3. Prototype auto-proposal: when the Apply hook surfaces the same correction memory more than N times, emit a draft guard (code plus calibration fixture) as a PR for human review, never auto-merged (change classes and review stay under ai-agents-change-control).

Falsifiable milestone

A correction observed once in a session auto-proposes a guard that survives the calibration replay and ships through review. Falsified if, after the pipeline exists, proposed guards systematically fail calibration (the #1887 result generalizes: guards cannot pay for themselves) or the loop only ever proposes guards for corrections that a human already hand-coded.

Anti-Patterns

Anti-pattern	Why it fails here	Do instead
Citing ADR-069/ADR-072 as accepted policy	Both are PROPOSED (as of 2026-07-30); ADR-072 has five unmet conditions to reach Accepted. ADR-068 did reach Accepted, so it is policy	Quote the status line before citing; treat the Proposed pair as research direction, not mandate
Shipping a detector or guard without replaying it on real history	#1989 M4 threshold could never fire; #1887 guards prevented 0/35 of their own fixes	Calibrate against roughly the last 5 real PRs first
Trusting vendor docs for harness behavior	Docs were wrong by omission in #2205 and #2290	Empirical probe with pinned version and negative control (ai-agents-empirical-probe-toolkit)
Claiming corpus counts or eval coverage from this file	Counts decay; this file is a snapshot	Re-run the Phase 1 verification commands
Starting a new capability outside the three programs without a gate	AGENTS.md requires the buy-vs-build Quick tier before spec for new capabilities	Run buy-vs-build-framework, then ai-agents-research-methodology
Publishing only positive results	Rejected fixes are load-bearing knowledge (#2230 pattern)	Record negative results in a retro or decision memory

Verification

Before acting on this skill's claims, or after editing it:

• Ran the Phase 1 re-verify commands; every status and count still matches, or this file was updated with new date stamps.
• Any program step you are about to start has its predicted outcome written down before the measurement runs.
• Everything you plan to cite as settled is Accepted or code-on-disk; all PROPOSED and UNVERIFIED labels preserved in your write-up.
• Your work routes through ai-agents-change-control gates (plugin version-field, adr-review, drift checks) rather than around them.

Provenance and Maintenance

Authored 2026-07-03, facts re-verified against the working tree on 2026-07-30. Retro-cited SHAs ddb76e0 and 01e76615a exist in this clone but are not reachable from main. Clone refs determine whether those objects exist, so verify ancestry before using git log. Prefer .agents/retrospective/ and .serena/memories/ for the reasoning behind a change, which commit messages rarely carry.

Sources and re-verification:

• ADR-069 thesis and status: .agents/architecture/ADR-069-context-corpus-is-the-product.md:2 (status: proposed), title at line 9. Re-verify: head -12 .agents/architecture/ADR-069-context-corpus-is-the-product.md.
• ADR-072 status, review verdict, five conditions, harness list: .agents/architecture/ADR-072-jtbd-plugin-architecture.md:3-20,119-131. Re-verify: sed -n '1,25p;119,131p' .agents/architecture/ADR-072-jtbd-plugin-architecture.md.
• ADR-068 status and #2295 measurements (3/197 kills, ~246 ms cold start, 40 shims): .agents/architecture/ADR-068-consolidated-hook-dispatcher.md. Re-verify: sed -n '1,10p' .agents/architecture/ADR-068-consolidated-hook-dispatcher.md; grep -n -A1 -e "Three of" -e "246" -e "N=40" .agents/architecture/ADR-068-consolidated-hook-dispatcher.md.
• Rule-activation eval mechanisms, judge dimensions, exit codes: scripts/eval/eval-rule-activation.py:1-40 docstring. Re-verify: sed -n '1,40p' scripts/eval/eval-rule-activation.py.
• FM-1 95.8% evidence: .agents/governance/FAILURE-MODES.md:44. Re-verify: grep -n "95.8" .agents/governance/FAILURE-MODES.md.
• Detect-Log-Graduate and explicit retrieval: the reflect skill, .claude/skills/memory/SKILL.md, and .claude/skills/memory-search/SKILL.md. Re-verify the deleted advisory hooks' absence with the Phase 1 test command.
• EVENT telemetry emitter and tier classifier: RETIRED. push_guard_base.py, every guard built on it, and the skill that classified guards into Budding/Growing/Mature/Proficient/Inert/Harmful tiers were all deleted under ADR-084 (issue #5154); no live file emits or consumes this schema. Re-verify the removal: ls .claude/hooks/PreToolUse/ (expect no push_guard_base.py or invoke_*_guard.py) and ls .claude/skills/ | grep guard-maturity (expect no output).
• Runtime contract test and anchoring validator: tests/build_scripts/test_generate_hooks_runtime_contract.py, scripts/validation/validate_hook_anchoring.py. Re-verify: ls tests/build_scripts/test_generate_hooks_runtime_contract.py scripts/validation/validate_hook_anchoring.py.
• Env-anchor decision memory: .serena/memories/decision-copilot-cli-hook-plugin-root-contract.md. Re-verify: ls .serena/memories/decision-copilot-cli-hook-plugin-root-contract.md.
• Counts (skill dirs, rules, retros, memories, scenario fixtures): Phase 1 command. Volatile; run the command, never quote a stored number.
• Incident claims (#2205, #2290, #1887, #1989, #2230): see ai-agents-failure-archaeology for evidence paths; do not re-litigate settled battles.

Unverified in this document (flagged inline): per-rule traffic data, automated EVENT consumer pipeline, Forgetful population status, issue #1345 and #1859 original text (cited via on-disk docstrings and ADR references only).