Research methodology

Research Methodology

Purpose

Investigation fails in two ways: presenting assumptions as facts, and researching forever instead of acting. This skill is the runbook for both — every claim gets an explicit epistemic status backed by a primary source, and every investigation has stated stopping criteria. The output is always a digest a decision-maker (human or agent) can act on without re-deriving the work.

When to Use / When NOT to Use

Use when:

• Answering "why does X happen?", "how does Y work?", "is claim Z true?" about any system.
• Investigating an unfamiliar tool, API, dependency, or protocol before building on it.
• Auditing statements someone else made (docs, comments, commit messages, a previous agent's report).
• Producing findings another agent or person will act on — the epistemic labels are the handoff contract.
• Model memory conflicts with observed behavior.

Do NOT use when:

• First contact with a whole codebase (structure, entry points, conventions) → load codebase-onboarding; return here for specific open questions it surfaces.
• Chasing a live defect with a reproducer → load debugging-playbook (hypothesis discipline overlaps, but that skill owns the reproduce–bisect–fix loop).
• Reconstructing how a past failure happened from history → load failure-archaeology.
• Mapping components and dependencies of a system → load architecture-analysis.
• The unknown is "should this code exist at all?" → load engineering-minimalism.
• The answer is already verified and you're stalling — stop researching, act (see step 8).

Core Methodology

1. Write the question and the decision it feeds

One sentence each: "Question: does X do Y?" and "Decision: if yes we do A, if no we do B." If no decision depends on the answer, the research is speculative — stop (and consider engineering-minimalism).

2. Form explicit hypotheses before looking

Write down what you expect to find and why. A hypothesis you didn't write down mutates silently into "what I always thought" after you see the evidence. Multiple competing hypotheses are better than one.

3. Rank your sources: code > docs > memory

• Primary: the code itself, git history, actual command output, actual API responses. These can be quoted.
• Secondary: README/docs/comments — statements of intent, which drift from behavior. Trust only after spot-checking against primary.
• Tertiary: your training memory. Never sufficient alone for any claim that matters; APIs, versions, and flags drift past any knowledge cutoff. Memory generates hypotheses; it never confirms them.

Source decision tree for any claim you are about to write down:

Claim about to be written
├── Did I observe it directly? (read the code, ran the command, saw output)
│     └── YES → label VERIFIED, cite the file/commit/command
├── Does it follow logically from verified facts?
│     └── YES → label INFERRED, state the reasoning chain
├── Do I merely expect it? (docs said so, memory says so, "usually true")
│     ├── Can I check it cheaply now? → CHECK IT, then relabel
│     └── Can't check now → label HYPOTHESIS or REQUIRES VERIFICATION
└── None of the above → do not write the claim

4. Seek disconfirming evidence

For each hypothesis, ask: "what would I see if this were FALSE?" — then look for that, specifically. Grep for the counter-example, read the error path not just the happy path, check the commit that removed something, not only the one that added it. Confirmation-only research is how wrong answers arrive with high confidence.

5. Label every claim with epistemic status

Four labels, defined:

• verified — directly observed against a primary source; citation attached (path, commit hash, command + output).
• inferred — logical consequence of verified facts; reasoning stated.
• hypothesis — plausible, believed, unchecked.
• requires verification — needed for the decision but not yet checked; blocking or non-blocking stated.

Never present an assumption as a fact. In prose, the parenthetical form is fine: "(hypothesis — requires verification)". Un-labeled claims default to hypothesis in the reader's mind — so label them yourself, honestly.

6. Date-stamp volatile findings

Anything that can drift — versions, commit tips, published package state, dist-tags, API behavior, config values — gets a stamp: "(as of YYYY-MM-DD)", filled with the actual verification date. Stable findings (an algorithm in a pinned file at a named commit) may cite the commit instead. A dated finding is re-checkable; an undated one is a future lie.

7. Structure findings into digest form

The deliverable is a digest, not a transcript:

1. Question + decision it feeds (from step 1)
2. Answer in 1–3 sentences, with overall confidence
3. Key findings, each with epistemic label + citation + date stamp
4. What was ruled out (disconfirmed hypotheses — this is where step 4's work shows)
5. Open items — each "requires verification" entry, with the exact command to close it
6. Recommended action

8. Know when to stop

Stop researching and act when ANY of:

• The decision from step 1 is now determined either way.
• Remaining unknowns are labeled, non-blocking, and have re-check commands attached.
• The last two search rounds produced no new relevant facts (diminishing returns).
• Cost of further research exceeds the cost of acting and being cheaply wrong (reversible action + labeled uncertainty beats another hour of reading).

Do NOT stop while a blocking claim is still labeled hypothesis — that is exactly the claim that must be verified or explicitly flagged to the decision-maker.

Frontier adoption — evaluating new tools, libraries, models, and techniques

A special case of steps 1–8 with its own gate. When the question is "should we adopt X?" (a new framework, model, library, or technique):

1. Name the incumbent. Every adoption question is a comparison. "Adopt X" really means "replace [current approach] with X" — even when the incumbent is "nothing / hand-rolled." If you cannot name what X replaces, return to step 1 of the core method: no decision depends on the answer.
2. Demand evidence at the right tier. Marketing pages and benchmark claims are tertiary. A maintained repo with recent commits, an issue tracker you read (open bugs ARE documentation), and a working spike you ran yourself are primary. Never adopt on secondary evidence alone.
3. Run a bounded spike. Time-box a minimal integration (hours, not days) against your real use case, not the tool's demo. The spike's output is a digest (step 7) with a recommendation, not production code.
4. Price the full cost. Adoption cost = integration + the dependency's transitive surface (dependency-management) + team/agent learning curve + exit cost if it dies. Compare against the incumbent's known cost, not against zero.
5. Default to the incumbent on a tie. Novelty is not a tiebreaker; switching costs are real and the incumbent's failure modes are already known (engineering-minimalism rung 1 applies: is the need real, today?).
6. Date-stamp the verdict. Frontier facts rot fastest of all — a "not ready" verdict on a fast-moving tool deserves a re-check date, not permanence.

Investigation checklist

• Question and dependent decision written (1 sentence each)
• Hypotheses written down before evidence-gathering
• Every claim traced to a primary source or labeled otherwise
• Disconfirming evidence actively sought for each hypothesis
• All four epistemic labels used honestly (an all-"verified" report is suspicious)
• Volatile findings date-stamped
• Digest produced (answer, findings, ruled-out, open items, action)
• Stopping criterion stated and met

Discovery Commands

Primary-source interrogation, repo-agnostic:

# What actually happened, in order (never trust a summary of history)
git log --oneline -30
git log --oneline --all --graph | head -40

# Verify a specific claim about a change: read the actual diff
git show <commit-hash>
git show <commit-hash> --stat

# When/why did this line come to exist?
git blame -L <start>,<end> <file>

# Find when a string appeared/disappeared anywhere in history
git log -S "search-string" --oneline

# Hunt the counter-example (disconfirming evidence)
grep -rn "the-thing-that-should-not-exist-if-hypothesis-true" .

# Does the claimed file/flag/config actually exist?
find . -name "<claimed-file>" -not -path "*/node_modules/*" -not -path "*/.git/*"
ls -la <claimed-path>

# What does the project SAY it does (secondary source, to be spot-checked)
find . -maxdepth 2 -iname "README*" -o -iname "*.md" | head

# Verify declared dependencies/versions against reality
grep -n "version\|dependencies" package.json pyproject.toml setup.py 2>/dev/null

Rule: any claim in your digest that cannot be regenerated by one of these command patterns is not "verified" — relabel it.

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
Confident answer turns out wrong	Memory presented as fact	Memory only generates hypotheses; verify against code (step 3)
Docs say X, code does Y, report said X	Secondary source trusted without spot-check	Code > docs; quote the code (step 3)
Only supporting evidence in the digest	Confirmation-seeking	Ask "what would I see if false?" and look for it (step 4)
Reader can't tell facts from guesses	Missing epistemic labels	Label every claim; unlabeled = hypothesis (step 5)
Finding was true in March, acted on in July, broke	No date stamp on volatile fact	Stamp "(as of DATE)"; attach re-check command (step 6)
40-message investigation, no conclusion	No stopping criteria	Decision-linked question + stop conditions upfront (steps 1, 8)
Handoff report forces reader to redo the work	Transcript instead of digest	Digest form: answer, findings, ruled-out, open items (step 7)
Hypothesis silently became "known" mid-thread	Hypotheses not written before looking	Write expectations down first (step 2)
Blocking unknown shipped as a footnote	"Requires verification" not triaged	Mark blocking vs non-blocking; never act past a blocking one (step 8)
Research on a question nobody needs answered	No dependent decision	If no decision changes, stop; consider engineering-minimalism (step 1)

Repository Examples

Case study: ruflo — the claim that drifted (as of 2026-07-04)

~/ruflo — pnpm monorepo publishing three npm package aliases at identical versions with coordinated dist-tags. The docs mark the ruflo alias dist-tag step "EASY TO FORGET" — a written acknowledgment that a claim like "all aliases are published consistently" is volatile and needs per-release re-verification, not memory. The slip actually happened: commit "fix: @claude-flow/browser peer dep, dist-tags, bump to alpha.3" is the primary-source evidence that the documented risk materialized. Lesson applied to method: registry state and dist-tags are tertiary-trust items — verify at the registry, date-stamp the finding, never carry it across sessions unverified.

Case study: ragit — verifying a dependency claim via history (as of 2026-07-04)

~/ragit — local RAG CLI. The claim "numpy must stay <2.0 here" is not folklore: it is verified by commits a1ad63ed "Pin chromadb to 0.4.x" and 0a06ae9b "Pin numpy for chromadb 0.4", pinning chromadb>=0.4.0,<0.5.0 + numpy<2.0 after a transitive break. Re-check command: cd ~/ragit && git show 0a06ae9b. This is the pattern: a pin is a finding with provenance — the hash is the citation. (Related follow-up lives in dependency-management.)

Case study: prism — commit lineage as primary source (as of 2026-07-04)

~/prism — the claim "security hardening exists and is tested" verifies directly against history: e264a72 → b7bbabb → 51b92c9 "security: add payload size limit, concurrency cap, repo name validation, Ollama timeout" → b011239 "tests: add security path coverage for signature, repo validation, size limit". Verified by git log --oneline in the repo, 2026-07-04. Contrast a properly labeled weaker claim from the same investigation set: ASVER's "Remove server build from package.json for static deployment" motive — strip what the deployment shape forbids — is partly inferred, and is recorded with that label rather than upgraded to fact.

Validation Criteria

You applied this skill correctly when:

1. Every claim in your output carries one of the four epistemic labels (or an obvious verified citation).
2. Each "verified" claim has a citation a reader can replay: path, hash, or command.
3. At least one hypothesis was actively tested for disconfirmation — and the digest says what was ruled out.
4. Volatile findings carry date stamps; each open item carries the exact command that would close it.
5. The digest answers the step-1 question directly, and the dependent decision can be made from it without re-research.
6. You stopped: the thread ends in an action or recommendation, not another search.

Provenance & Maintenance

• Sources: ~/ruflo, ~/ragit, ~/prism, ~/asver — investigated 2026-07-04. Methodology is general; the repos supply worked examples of provenance discipline. The ASVER motive is explicitly (partly inferred), preserved as a live demonstration of honest labeling.
• Assumptions: commit hashes (a1ad63ed, 0a06ae9b, e264a72, b7bbabb, 51b92c9, b011239) and quoted commit messages are point-in-time facts from those repos' histories. The ruflo dist-tag commit is cited by message, not hash — locating its hash (requires verification): cd ~/ruflo && git log --oneline --all | grep -i "dist-tag".
• Re-verification commands:

  cd ~/ragit && git show --stat 0a06ae9b && grep -n "numpy\|chromadb" pyproject.toml setup.py 2>/dev/null
  cd ~/prism && git log --oneline -6
  cd ~/ruflo && git log --oneline --all | grep -i "dist-tag\|alpha.3"
  

• Likely to drift: npm registry/dist-tag state (per release), dependency pins (whenever chromadb/numpy move), commit tips, the ASVER inference (may be confirmed or refuted by its owner).
• Maintenance checklist:
  • Re-run re-verification commands; re-stamp dates.
  • Upgrade or delete the ASVER example once its motive is confirmed.
  • Confirm cross-referenced skills (codebase-onboarding, debugging-playbook, failure-archaeology, architecture-analysis, engineering-minimalism, dependency-management) still exist under those directory names.