Research critique

Research Review

Overview

Two operating modes depending on input:

• Claim/Evidence Evaluation — assess a claim, study finding, or piece of evidence using critical thinking frameworks
• Manuscript Review — formally review a paper, preprint, or presentation using structured peer review workflow

Both modes draw on the same shared frameworks: bias detection, statistical evaluation, and evidence quality. Mode is detected from context. If ambiguous, ask.

---

Mode 1: Claim/Evidence Evaluation

Use when given a claim, result, or study to assess rather than a full manuscript to formally review.

Claim Evaluation Process

1. Identify the claim — What exactly is asserted? Causal, associational, or descriptive? How strong is the claim (proven / likely / suggested / possible)?
2. Assess the evidence — What evidence is provided? Direct or indirect? Sufficient for the strength of claim? Are alternative explanations ruled out?
3. Check logical connection — Do conclusions follow from data? Logical leaps? Correlational data supporting causal claims? Limitations acknowledged?
4. Evaluate proportionality — Is confidence proportional to evidence? Are hedging words used appropriately? Is speculation labeled?
5. Check for overgeneralization — Do claims extend beyond the sample? Population restrictions acknowledged? Context-dependence recognized?
6. Red flags — Causal language from correlational data; "proves" or absolute certainty; cherry-picked citations; ignored contradictions; extrapolation beyond data.

Output format for claim evaluation:

1. Summary — what was evaluated
2. Strengths — what is well-supported
3. Concerns — organized by severity (critical / important / minor)
4. Recommendations — actionable suggestions
5. Overall Assessment — what conclusions the evidence actually supports

---

Mode 2: Manuscript Review

Use when given a full paper, manuscript, preprint, or presentation to formally review.

Stage 1: Initial Assessment

• What is the central research question or hypothesis?
• What are the main findings and conclusions?
• Is the work scientifically sound and significant?
• Are there immediate major flaws that preclude publication?

Output: 2-3 sentence summary capturing the manuscript's essence and initial impression.

Stage 2: Section-by-Section Review

Abstract and Title

• Accuracy: Does the abstract reflect actual content and conclusions?
• Clarity: Is the title specific, accurate, informative?
• Completeness: Are key findings and methods summarized?

Introduction

• Context: Is background adequate and current?
• Rationale: Is the research question clearly motivated?
• Novelty: Is originality and significance articulated?
• Objectives: Are aims/hypotheses clearly stated?

Methods

• Reproducibility: Can another researcher replicate from this description?
• Rigor: Are methods appropriate for the research question?
• Detail: Are protocols, reagents, parameters sufficiently described?
• Ethics: Are approvals, consent, and data handling documented?
• Statistics: Are statistical methods appropriate, described, justified?
• Validation: Are controls, replicates, and validation adequate?

Critical elements: sample sizes, power calculations, randomization, blinding, inclusion/exclusion criteria, software versions, multiple comparison correction.

Results

• Presentation: Logical, clear, without over-interpretation?
• Statistics: Effect sizes, CIs, p-values properly reported?
• Completeness: Are negative results included?
• Common issues: selective reporting, missing error bars, overfitting, batch effects, missing controls.

Discussion

• Interpretation: Conclusions supported by data?
• Limitations: Acknowledged and discussed?
• Context: Findings placed within existing literature?
• Red flags: Overstated conclusions; ignored contradictions; causal claims from correlational data; mechanistic claims without mechanistic evidence.

References

• Completeness, currency, balance (contrary viewpoints cited), accuracy, appropriate self-citation.

Stage 3: Reproducibility and Transparency

Data Availability

• Raw data deposited in appropriate repositories?
• Accession numbers provided?
• Data sharing restrictions justified?

Code and Materials

• Analysis code available (GitHub, Zenodo)?
• Unique materials available or described for recreation?

Reporting Standards

• Does the manuscript follow discipline-specific guidelines: CONSORT (RCTs), PRISMA (systematic reviews), ARRIVE (animal research), STROBE (observational studies), MIAME/MINSEQE (omics)?
• Are all elements of the relevant checklist addressed?

Stage 4: Figure and Data Presentation

Quality checks

• High resolution, clearly labeled axes with units?
• Error bars defined (SD, SEM, CI)?
• Color schemes colorblind-accessible?
• Scale bars included for images?
• Figures stand alone with legends?

Integrity checks

• Signs of image manipulation (duplication, splicing)?
• Western blots/gels appropriately presented?
• Representative images truly representative?
• All conditions shown (no selective presentation)?

Stage 5: Ethical Considerations

Human subjects — IRB approval documented; informed consent described; vulnerable populations protected; conflicts of interest disclosed.

Animal research — IACUC approval documented; 3Rs (replacement, reduction, refinement) considered; procedures humane and justified.

Research integrity — Data fabrication/falsification concerns; authorship appropriate; competing interests disclosed; funding disclosed; plagiarism or duplicate publication concerns.

Stage 6: Special Considerations by Manuscript Type

Original research — Emphasize rigor, reproducibility, novelty; verify conclusions are data-driven; check complete methods and appropriate controls.

Reviews and meta-analyses — Comprehensiveness of literature coverage; search strategy and inclusion/exclusion criteria; systematic approach; critical analysis vs. mere summarization; for meta-analyses, assess statistical approach and heterogeneity.

Methods papers — Validation and comparison to existing methods; reproducibility; sufficient detail for implementation.

Preprints — Recognize pre-peer-review status; still apply rigorous standards; constructive feedback to help before journal submission.

Presentations and slide decks — NEVER read the PDF directly. Convert to images first using any PDF-to-image tool available (e.g., pdftoppm, ImageMagick's convert, or a Python script using pdf2image). Inspect each slide image sequentially. Document issues with specific slide numbers.

Presentation-specific criteria: text ≥18pt; contrast ≥4.5:1; no overflow or element overlaps; one main idea per slide; ≤6 bullets per slide; results dominate (40-50%); ~1 slide per minute; narrative arc (intro → methods → results → discussion); citations in intro and discussion.

---

Shared Frameworks

These apply in both modes.

Bias Detection

Cognitive biases (researcher)

• Confirmation bias: Only supporting findings highlighted?
• HARKing: Hypotheses stated a priori or formed after seeing results?
• Publication bias: Negative results missing from literature?
• Cherry-picking: Evidence selectively reported?
• Check for preregistration and analysis plan transparency.

Selection biases

• Sampling bias: Is sample representative of target population?
• Volunteer bias: Do participants self-select systematically?
• Attrition bias: Is dropout differential between groups?
• Survivorship bias: Only "survivors" visible in sample?

Measurement biases

• Observer bias: Could expectations influence observations?
• Recall bias: Are retrospective reports systematically inaccurate?
• Social desirability: Responses biased toward acceptability?
• Instrument bias: Do measurement tools systematically err?

Analysis biases

• P-hacking: Multiple analyses until significance emerged?
• Outcome switching: Non-significant outcomes replaced with significant ones?
• Selective reporting: All planned analyses reported?
• Subgroup fishing: Subgroup analyses without correction?
• Check study registration and compare to published outcomes.

Confounding

• What variables could affect both exposure and outcome?
• Were confounders measured and controlled?
• Could unmeasured confounding explain findings?
• Plausible alternative explanations?

Statistical Analysis Evaluation

1. Sample size and power — A priori power analysis conducted? Adequate for detecting meaningful effects? Underpowered studies (common) raise flags for inflated effect sizes in significant results.
2. Statistical tests — Appropriate for data type and distribution? Assumptions checked? Parametric vs. non-parametric choice justified? Analysis matched to design (paired vs. independent)?
3. Multiple comparisons — Multiple hypotheses tested? Correction applied (Bonferroni, FDR)? Primary outcomes distinguished from exploratory? Could findings be false positives?
4. P-value interpretation — Interpreted correctly? Non-significance incorrectly treated as "no effect"? Statistical significance conflated with practical importance? Suspicious clustering just below .05?
5. Effect sizes and CIs — Effect sizes reported alongside significance? CIs show precision? Effect size meaningful in practical terms?
6. Missing data — How much missing? Mechanism considered (MCAR, MAR, MNAR)? Handling method (deletion, imputation, maximum likelihood) appropriate?
7. Regression and modeling — Overfitting? Extrapolation outside data range? Multicollinearity addressed? Model assumptions checked?
8. Common pitfalls — Correlation as causation; regression to the mean ignored; base rate neglect; Texas sharpshooter; Simpson's paradox.

Evidence Quality Assessment

Study design hierarchy (highest to lowest for intervention effects):

1. Systematic reviews / meta-analyses
2. Randomized controlled trials
3. Cohort studies
4. Case-control studies
5. Cross-sectional studies
6. Case series / reports
7. Expert opinion

Higher-level designs are not always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.

GRADE framework

• Start: RCT = high; observational = low
• Downgrade for: risk of bias; inconsistency across studies; indirectness (wrong population/intervention/outcome); imprecision (wide CIs, small samples); publication bias
• Upgrade for: large effect sizes; dose-response relationships; confounders would reduce (not increase) the effect

Convergence of evidence

• Stronger: multiple independent replications; different research groups and settings; different methodologies converging; mechanistic and empirical evidence aligned
• Weaker: single study or group; contradictory findings; publication bias evident; no replication attempts

Contextual factors — Biological/theoretical plausibility; consistency with established knowledge; temporality (cause precedes effect); specificity; strength of association.

Logical Fallacy Identification

Causation fallacies — Post hoc ergo propter hoc; correlation = causation; reverse causation; single cause fallacy.

Generalization fallacies — Hasty generalization; anecdotal fallacy; cherry-picking; ecological fallacy (group patterns applied to individuals).

Authority and source fallacies — Appeal to authority (without evidence); ad hominem; genetic fallacy; appeal to nature.

Statistical fallacies — Base rate neglect; Texas sharpshooter; multiple comparisons not corrected; prosecutor's fallacy (confusing P(E|H) with P(H|E)).

Structural fallacies — False dichotomy; moving goalposts; begging the question; straw man.

Science-specific fallacies — Galileo gambit; argument from ignorance; nirvana fallacy; unfalsifiability.

When identifying fallacies: name the specific fallacy; explain why the reasoning is flawed; identify what evidence would be needed for valid inference; note that fallacious reasoning doesn't prove the conclusion false — just that this argument doesn't support it.

---

Manuscript Review Report Format

Summary Statement

1-2 paragraphs: brief synopsis; overall recommendation (accept / minor revisions / major revisions / reject); 2-3 key strengths; 2-3 key weaknesses; bottom-line assessment.

Major Comments

Numbered sequentially. Each entry: (1) state the issue, (2) explain why it's problematic, (3) suggest specific solutions, (4) indicate if essential for publication.

Typical major comments: fundamental methodological flaws; inappropriate statistical analyses; unsupported conclusions; missing critical controls; serious reproducibility concerns; ethical concerns.

Minor Comments

Numbered sequentially with specific location (section, paragraph, figure). Typical: unclear figure labels, missing methodological details, typos, suggestions for improved data presentation, requests for clarification.

Questions for Authors

Methodological details that are unclear; seemingly contradictory results; missing information needed to evaluate; requests for additional data or analyses.

---

Tone and Approach

• Constructive: frame criticism as opportunities for improvement
• Specific: concrete examples and actionable suggestions
• Balanced: acknowledge strengths as well as weaknesses
• Respectful: authors have invested significant effort
• Objective: focus on the science, not the scientists

Avoid: personal attacks; sarcasm; vague criticism without examples; demanding unnecessary experiments beyond scope; applying stricter standards to results you dislike.

---

Research Design Guidance

When helping plan or improve a study:

1. Research question — Specific, answerable, falsifiable? Addresses a gap? Feasible? Variables operationally defined?
2. Design selection — Matched to question (causal → experimental; associational → observational)? Feasibility and ethics considered?
3. Bias minimization — Randomization where possible; blinding at all feasible levels; confounders identified and controlled; procedures standardized.
4. Sample planning — A priori power analysis; account for attrition; clear inclusion/exclusion criteria; representativeness.
5. Measurement strategy — Validated, reliable instruments; objective measures preferred; multiple measures of key constructs; inter-rater reliability procedures.
6. Analysis planning — Prespecify all hypotheses and analyses; designate primary outcome; plan assumption checks; specify missing data handling; plan to report effect sizes and CIs; consider multiple comparison corrections.
7. Transparency — Preregister study and analysis plan; use reporting guidelines (CONSORT, STROBE, PRISMA); plan to report all outcomes; distinguish confirmatory from exploratory; commit to data/code sharing.