Research and synthesis

Skill 09: Effective Research & Information Synthesis

Purpose: Find reliable information, judge its trustworthiness, and combine it into a grounded answer, without inventing facts. Use when: You need knowledge you don't have: an unfamiliar API, a design decision, a factual question, "how do others solve this," a comparison of options. Don't use when: You already have reliable, sufficient knowledge, don't over-research a solved question. And don't research your way out of a task that just needs doing.

---

Why this matters

Weak reasoners have two opposite research failures. One is hallucination: confidently stating things that aren't true because it feels like they should be, inventing an API method, a statistic, a library that doesn't exist. The other is shallow single-sourcing: grabbing the first result and treating it as gospel. Both produce work built on sand. Good research is the habit of grounding claims in evidence, checking that evidence, and being honest about the boundary between what you know and what you're guessing.

The core principle

Ground every load-bearing claim in a source you actually checked, and clearly separate what you know from what you assume. The most dangerous output is a confident falsehood, because it's acted on as if true. "I don't know yet, let me verify" is a strength, not a weakness.

---

The research process

Step 1: Define the exact question

Write the specific thing you need to know, as a question. "Does library X support async in version ≥2.0?" is answerable; "learn about X" is a rabbit hole. A sharp question tells you when you're done.

Step 2: Decide how sure you need to be

Match rigor to stakes:

Stakes	Rigor
Reversible, low-cost	One good source; proceed and correct later.
Shapes the design	Two independent sources or a direct test.
Irreversible / high-cost / safety / security	Primary source + independent confirmation + your own verification.

Step 3: Go to the right source

Prefer sources by reliability, roughly:

1. Primary & authoritative: official docs, the actual source code, the spec, the real data, a direct experiment.
2. Reputable secondary: well-regarded references, peer-reviewed or widely-corroborated material.
3. Community / anecdotal: forums, blog posts, answers. Useful for leads and gotchas; verify before relying.
4. Your own memory / intuition: a hypothesis to check, never a citation. Especially for exact names, numbers, versions, and syntax, which memory reconstructs wrongly.

When you can, prefer testing over reading. The most reliable source for "does this actually work" is running it. A three-line experiment beats a paragraph of documentation you might be misreading.

Step 4: Verify before trusting

• Cross-check anything important against a second, independent source. Two sources copying each other is one source.
• Check the date. Software, prices, and best practices go stale. Confirm the information matches the version/era you're in.
• Confirm it fits your case. A correct answer to a slightly different question is a wrong answer to yours. Note the differences (the "disanalogy" from first-principles-reasoning).

Step 5: Synthesize, don't just collect

Combine sources into an answer to your question:

• Reconcile disagreements: when sources conflict, figure out why (different versions? different contexts? one is wrong?). Don't average them; understand them.
• Distinguish consensus from outlier. One contrarian post against ten agreeing sources is a flag to investigate, not equal weight.
• State the answer with its confidence and its basis: "X supports async as of v2.1 (per the official changelog and confirmed by a quick test). Below 2.0 it does not."

Step 6: Mark the boundary of what you know

Explicitly separate:

• Verified: checked against a reliable source or test.
• Inferred: reasoned from what you verified, but not directly confirmed.
• Unknown / assumed: you're proceeding on an assumption; flag it so it can be corrected.

Never present inferred or assumed as verified. This one habit prevents most research-driven errors.

---

Anti-hallucination rules

These are non-negotiable when accuracy matters:

1. Never invent specifics. API names, function signatures, flags, version numbers, statistics, citations, file paths: if you're not sure, look it up or say you're unsure. A plausible-sounding method name that doesn't exist is worse than "let me check the docs."
2. "I think" is a signal to verify. If you catch yourself hedging internally, that's the moment to check, not to state it confidently anyway.
3. Distinguish "I recall" from "I confirmed." Memory is a starting hypothesis for exact details, not a source.
4. If you can't verify and must proceed, say so. "Assuming the API is unchanged since my knowledge cutoff: verify this" is honest and actionable.
5. A cited source you didn't read is not a citation. Don't attribute claims to sources you haven't actually checked.

---

Worked example

Question: "What's the cheapest way to store 500 GB of rarely-accessed backups on our cloud provider?"

Weak research: Recall that "cold storage is cheap," pick a service name from memory, quote a price that feels right. → The service name is slightly wrong, the price is two years stale, and it charges heavily for retrieval, which matters for backups.

Good research:

1. Question: Cheapest storage class for 500 GB, write-rarely, read-rarely, on this specific provider, including retrieval cost.
2. Rigor: This affects a recurring bill → design-shaping → two sources + check pricing page directly.
3. Source: The provider's current official pricing page (primary), not a blog from 2022.
4. Verify: Cross-check the retrieval-cost gotcha, since backups occasionally are restored, confirm the archive tier's retrieval fees and minimum storage duration.
5. Synthesize: "Archive tier is cheapest per-GB for storage, but has a 180-day minimum and per-GB retrieval fees; for backups restored rarely that's still cheapest overall. (Per current pricing page, checked today.)"
6. Boundary: "Verified: storage and retrieval prices as of today. Assumed: restore frequency < 1/quarter, confirm with the team, it flips the math if higher."

The good version is correct, current, fits the real use case, and flags the one assumption that could change the answer.

---

Common failure modes

Failure	Fix
Hallucinating specifics	Look it up; never invent names/numbers/citations.
Single-sourcing important claims	Cross-check independently.
Stale information	Check dates; confirm version relevance.
Answering a nearby question	Confirm the source fits your case; note differences.
Collecting without synthesizing	Reconcile and answer your question.
Presenting guesses as facts	Label verified / inferred / assumed.
Reading when you could test	Run a small experiment for "does it work."
Over-researching	Match rigor to stakes; stop when the question is answered.

Red flags: stop and verify

• You're about to state an exact name, number, version, or citation from memory.
• You've only looked at one source for something that matters.
• The information could be outdated and you haven't checked when it's from.
• You can't tell whether the source's situation matches yours.
• You're presenting a guess without labeling it as one.

Definition of done for this skill

• The research question was specific and now answered.
• Rigor matched the stakes.
• Load-bearing claims are grounded in checked, appropriately-authoritative sources.
• Important claims were cross-checked and date-checked.
• The answer distinguishes verified / inferred / assumed.
• No invented specifics.

See also

• first-principles-reasoning, applying and adapting what you find.
• documentation-and-knowledge-capture, recording what you learned so you don't re-research it.
• self-verification, the "prefer testing" habit.