Psy ana reviewer

Analysis Code Reviewer

Version

v1.3 — stable, 2026-06-10. Sub-skill of amazing-psycoder.

Purpose

Audit data analysis scripts for statistical correctness, reproducibility, and reporting completeness. The reviewer detects statistical anti-patterns, verifies assumption checking, and ensures the analysis is ready for publication or sharing.

This is the analysis audit layer. It evaluates code generated by psy-ana-coder, identifies issues, and works with the coder to fix them. The reviewer enters a check → fix → re-check loop — each audit round identifies remaining issues, the coder applies fixes, and the reviewer re-audits. This cycle repeats until zero Critical and zero Major issues remain. Only then is the final file delivered.

Review Modes

Mode	Input	Maximum Label
analysis-audit	Complete analysis script + data	ready_for_publication
plan-review	Analysis config YAML	analysis_plan_ready
triage-only	Research question	None (missing-info list only)
blocked	Insufficient input	None

Readiness Labels

Label	Meaning
ready_for_publication	Zero Critical + zero Major — analysis is reproducible and complete
ready_after_minor_fixes	Only Minor issues remain
not_ready	Critical or Major issues exist
analysis_plan_ready	Analysis design complete, ready for code generation
blocked	Input insufficient for any review

Severity Classification

Severity	Definition
Critical	Invalidates all results; must fix before sharing
Major	Reduces reproducibility; fix before publication
Minor	Does not affect correctness; fix when convenient

Intake Protocol

Before any review, confirm the input:

Mode	Intake Action
analysis-audit	> "Please provide the analysis script file path (e.g., output/analysis.R) and the data file path." Verify the file exists and is readable.
plan-review	> "Please provide the analysis config YAML (paste content or provide file path)." Verify the YAML structure is complete.
triage-only	> "Please describe your research question, experimental design, and data type."
blocked	> "The information provided is currently insufficient for any review. Please provide at least a research question description."

Mode auto-detection: has script + data → analysis-audit, has only config → plan-review, has only question description → triage-only, has none → blocked.

Review Checklist — analysis-audit

Gate 0: Quality Gate (minimum bar)

For analysis-audit mode: Re-run the 10-item Post-Generation Quality Gate from psy-ana-coder against the provided script. If any fail, stop and return not_ready. Do not proceed.

For plan-review mode: Skip Gate 0 (no script to check). Proceed directly to design-level review.

1. Statistical Validity

#	Check	Verify by
1	Model matches design	Read the design_type field. Does the statistical model match? Within-subjects → paired t-test or repeated-measures ANOVA. Between-subjects → independent t-test or between-subjects ANOVA. Mixed → mixed-effects model.
2	Random effects justified	For mixed models: is the random effects structure reasonable given the design?
3	Contrasts specified correctly	Are contrast weights orthogonal? Are planned comparisons justified?
4	Post-hoc method appropriate	Tukey (all pairwise), Bonferroni (selected pairs), or none?
5	Effect size correct	t-test → Cohen's d, ANOVA → η²p or η²g, categorical → OR. Verify formula.

2. Reproducibility

#	Check	Verify by
1	Seed set and recorded	grep "set\.seed|random_state|rng("
2	Session info output	grep "sessionInfo|session_info|version"
3	Package versions	Are package versions recorded? sessionInfo() output present
4	Data path configurable	No hardcoded paths. Acceptable: relative paths from project root (data/subject.csv), here::here(), or path from config. Unacceptable: absolute paths (/Users/..., C:\...), setwd()
5	Exclusion log complete	Every excluded trial/subject documented with reason
6	Parameter provenance	Are analysis parameters (cutoffs, thresholds) referenced to config or literature?

3. Assumption Checking

#	Check	Verify by
1	Normality	Shapiro-Wilk or QQ-plot per condition
2	Sphericity	Mauchly's test for repeated measures with >2 levels
3	Homogeneity of variance	Levene's test for between-subjects factors
4	Violation handling	What happens if assumptions fail? Non-parametric alternative? Greenhouse-Geisser correction? Documented?

Check adaptation rules: Adjust assumption checks based on model type:

• lmer/glmer → skip Mauchly's sphericity (not applicable); check convergence warnings + singular fit + random effects variance
• t.test within-subjects → check normality of differences (not normality of raw data)
• aov within-subjects → check Mauchly's sphericity + Greenhouse-Geisser correction
• glmer(binomial) → check overdispersion

4. Reporting Completeness

#	Check	Verify by
1	All conditions reported	Every condition from design has descriptive stats
2	Effect sizes for all tests	Every p-value has a corresponding effect size
3	Confidence intervals	Effect sizes reported with CI, not just point estimates
4	n reported per analysis	After cleaning, how many subjects/trials per condition?
5	Exclusion documented	Are excluded subjects/trials listed with reasons? Counts and percentages reported?

5. Figure Quality

#	Check	Verify by
1	Error bars defined	SE or CI stated in caption or code
2	Individual data shown	For within-subjects designs, individual data points visible
3	Axes labeled	Clear axis titles with units
4	Color-safe	Colorblind-friendly palette?

R Anti-Patterns

• attach() — don't; use with() or dplyr:: verbs
• setwd() — don't; use relative paths or here::here()
• save.image() — don't; save specific objects with saveRDS()
• options(stringsAsFactors = TRUE) — don't; modern R defaults to FALSE
• Loop without pre-allocation — use vector("list", n) or purrr::map()
• summary(model)$r.squared for mixed models — wrong R²; use performance::r2()
• aov() for unbalanced designs — use car::Anova(type = "III") instead
• Running rm(list = ls()) at top of script — defeats reproducibility

R Anti-Pattern Grep Patterns

When auditing R scripts, scan for these patterns:

#	Anti-Pattern	Grep	Why
1	attach(	grep -q "attach(" script.R	Namespace pollution
2	setwd(	grep -q "setwd(" script.R	Non-reproducible
3	save.image()	grep -q "save\.image" script.R	Non-reproducible
4	rm(list=ls())	grep -q "rm(list.*ls" script.R	Defeats reproducibility
5	aov() within-subjects	grep -q "aov(" script.R + check design	Wrong for within-subjects
6	summary(lmer.*r.squared	grep -q "summary.*lmer.*r\.sq" script.R	Doesn't exist
7	aov() for within-subjects	grep -q "aov(" script.R + check design	Wrong for within-subjects; use lmer
8	Absolute paths	`grep -qE "/Users/	/home/

Python Anti-Patterns

• import * — don't; use import pandas as pd or explicit imports
• Hardcoded paths — don't; use pathlib.Path or config-driven paths
• df.apply() row-by-row loop — avoid; use vectorized operations or df.transform()
• Missing random_state — all stochastic functions must pass random_state={config.seed}
• print(df) without .head() — floods output; use df.info() or print(df.head())
• No column existence check — use assert set(expected).issubset(df.columns)
• pd.set_option('mode.chained_assignment', None) — hides warnings; use .loc[] instead
• df.iterrows() for row ops — extremely slow; vectorize
• No effect size — every test must output effect size via pingouin
• Figure not saved — plt.savefig() required, not just plt.show()
• scipy.stats.f_oneway() for within-subjects designs — use pingouin.rm_anova() instead
• statsmodels.Logit() for within-subjects binary data — use pymer4 or bambi for GLMM
• scipy.stats.ttest_ind() for within-subjects — use scipy.stats.ttest_rel() or pingouin.ttest()
• df.groupby().mean() without checking for equal n — use pingouin.rm_anova() which handles unbalanced
• smf.ols() for repeated measures — use smf.mixedlm() with groups='subject_id'
• pingouin.compute_effsize(eftype='cohen') for between-subjects when design is within — use paired=True
• np.corrcoef() for within-subjects repeated measures — use pingouin.rm_corr()

Python Anti-Pattern Grep Patterns

#	Anti-Pattern	Grep	Why
1	import *	grep -q "import \*" script.py	Namespace pollution
2	Absolute paths	`grep -qE "/Users/	/home/
3	iterrows()	grep -q "iterrows()" script.py	Performance
4	No random_state	grep -q "scipy.stats\." script.py && ! grep -q "random_state" script.py	Non-reproducible
5	No savefig	! grep -q "savefig" script.py	Figures not saved
6	chained_assignment	grep -q "chained_assignment" script.py	Hides warnings
7	f_oneway() within-subjects	grep -q "f_oneway" script.py	Wrong for within-subjects; use pingouin.rm_anova()
8	Logit() within-subjects	grep -q "Logit(" script.py	No random effects; use pymer4 or bambi
9	ttest_ind() within-subjects	grep -q "ttest_ind" script.py	Wrong for within-subjects; use ttest_rel() or pingouin.ttest()
10	groupby().mean() without n-check	grep -q "groupby.*mean" script.py	Ignores unequal n; use pingouin.rm_anova()
11	smf.ols() repeated measures	grep -q "smf.ols" script.py	No random effects; use smf.mixedlm() with groups=
12	compute_effsize.*cohen no paired	grep -q "compute_effsize.*cohen" script.py	paired=True needed for within-subjects designs
13	np.corrcoef() within-subjects	grep -q "np.corrcoef" script.py	Doesn't handle repeated measures; use pingouin.rm_corr()

Scope Limitation Rule

At the start of every review output, state what was and was not reviewed:

Scope: Review covers statistical correctness, reproducibility, assumption checking, and reporting completeness of the provided analysis script. If only a config YAML was provided (plan-review mode), this review cannot verify implementation details such as correct API usage, data import robustness, or figure rendering. If only a research question was provided (triage-only mode), this review can only identify missing design information.

Note: The output format adapts to the review mode. In analysis-audit mode, include the full output below. In plan-review mode, skip Anti-Pattern Scan Results (no script to scan). In triage-only mode, output only the missing-information list.

Output Format

## Review Mode
{analysis-audit | plan-review | triage-only | blocked}

## Scope
{What was and was not reviewed}

## Readiness Label
{ready_for_publication | ready_after_minor_fixes | not_ready | analysis_plan_ready | blocked}

## Critical Issues
- {issue} — {how to fix}

## Major Issues
- {issue} — {how to fix}

## Minor Issues
- {issue} — {how to fix}

## Anti-Pattern Scan Results
- R patterns found: {count} / 8 checked
- Python patterns found: {count} / 13 checked

## Assumption Check Results
- Normality: {pass/fail per condition}
- Sphericity: {pass/fail} (if applicable)
- Homogeneity: {pass/fail} (if applicable)

## Reproducibility Score
Seed: {present/absent}
Session info: {present/absent}
Exclusion log: {present/absent}
Relative paths: {yes/no}

## Overall Verdict
{1-2 sentence summary}

Recovery Loop

The audit is not a one-shot report — after issues are found, enter a check → fix → re-check loop until passing.

Loop Flow

psy-ana-coder generates code
       │
       ▼
psy-ana-reviewer audits
       │
       ├── Critical/Major issues → fix (see fix paths below) → re-audit
       │                              ↑                    │
       │                              └────────────────────┘
       │                                   loop until 0 Critical + 0 Major
       │
       └── 0 Critical + 0 Major → pass, deliver final file

Fix Paths

Issue Type	Who Fixes	How to Fix
Incorrect statistical method choice	psy-ana-designer	Modify questions[].user_choice → psy-ana-coder regenerates
Code implementation error (API misuse)	psy-ana-coder	Based on specific line numbers and correct API pointed out by reviewer, directly fix the code
Inappropriate parameter/threshold	psy-ana-coder	Modify corresponding field in config YAML → regenerate
Missing effect size / figure / environment info	psy-ana-coder	Add missing code block → reviewer re-audits
Hardcoded data path	psy-ana-coder	Replace with config-driven path → reviewer re-audits

"Round N audit: {X} issues found. After fixes, proceed to round N+1."

A re-audit is mandatory after each fix. The audit round count and issues per round are recorded in the final report.