Reviewer skill

Reviewer

Reviewer is a bundled academic citation-review workflow. Given one target paper, it can fetch citing papers from OpenAlex, classify whether each citing paper appears to use the target work, build a review-writing prompt, and optionally write the review using either the agent running this skill or an external cloud LLM.

Reviewer execution code upstream: https://github.com/nmelanitis/Reviewer/tree/main

Before Running

Start each new run by asking where the workflow should stop:

Where should Reviewer stop: fetch papers, classify papers, or write the review?

Default: write the review.

Then ask:

Do you want to set Reviewer options interactively, or use defaults?

Always ask for the target paper DOI unless the user already provided it. If the user chooses defaults, use these defaults:

• Full texts: yes, using open-access PDFs when available.
• Classification method: LLM.
• LLM execution: internal agent, not external cloud/API, unless the user explicitly chooses a provider.
• Review length: left unspecified.
• Final output: write the review, not only generate the prompt.
• Papers included in the review: yes papers only.
• References and bibliography: include paper references and a bibliography at the end, as in a scientific paper.
• Stop point: write the review.

If the user chooses interactive setup, ask for at least:

• Target paper identifier: DOI, DOI URL, or OpenAlex work ID.
• Where to stop: fetch papers, classify papers, or write the review. Default: write the review.
• Whether to use full texts. Default: yes.
• How to classify papers: regex or LLM. Default: LLM.
• Whether LLM steps should use the internal agent or an external cloud/API provider. Default: internal agent.
• Review length in pages, or leave unspecified. Default: unspecified.
• Whether to include in-text paper references and an end bibliography. Default: yes.
• Whether to write the review or only generate the review prompt. Default: write the review.
• Which papers to include: yes, yes and dont_know, or all. Default: yes.

Also clarify or infer these inputs as needed:

• Target aliases: acronyms, method names, spelling variants, and common short names that citing papers may use.
• Output folder name. Default: Papername_out, where Papername is the target paper title truncated to 20 characters and made filesystem-safe, for example Masked_Autoencoders_out.
• Classification route: metadata regex, generated regex, internal-agent LLM, or external cloud/API LLM.
• Evidence route: abstract-only or open-PDF full text.
• Cloud provider/model, only when the user wants external cloud/API LLM steps.
• Cost/privacy constraints before sending abstracts, full text, or prompts to a cloud provider.

Use a small --max-results or --limit smoke test before large cloud/API runs or very large internal-agent classification passes.

Resource Map

• src/: executable Reviewer pipeline scripts.
• docs/README.md: full command reference and examples.
• docs/ExecutionDiagram.md: choose between regex, direct LLM, abstract-only, and full-text routes.
• docs/Classify-papers-as-relevant-README.md: details for regex-context generation, generated regexes, and direct LLM classification.
• docs/Control-Review-size.README.md: prompt and output size controls.
• docs/paper_usage_regex_prompt.md: instruction prompt for generated-regex workflows.
• examples/: development examples for the MAE paper; read only when an example run pattern is useful.

Setup

Install the local dependencies in the environment used to run the scripts:

pip install pandas pypdf PyPDF2 PyMuPDF

For cloud LLM support, install only the providers needed:

pip install openai anthropic google-genai

Provider keys can be exported in the shell or stored in a local .env copied from .env.example. Never commit real API keys.

The internal-agent LLM path does not require provider SDKs or API keys.

Preferred Workflow

Prefer running individual scripts from the skill root with explicit input and output paths. This keeps generated artifacts out of the bundled src/ code and works after the code was reorganized into src/.

The preferred LLM execution path is the internal-agent path: the agent running this skill uses preserved prompt and evidence files to generate regexes, classify papers, and write the review without calling external LLM APIs. Use the external cloud/API path only when the user explicitly chooses it.

Create one output directory for the run. Name it Papername_out, where Papername is the target paper title truncated to 20 characters and made filesystem-safe. In examples, OUT_DIR means that folder. All generated artifacts must stay inside this folder: CSV files, metadata JSON, prompt Markdown files, downloaded PDFs, extracted full text, generated regexes, raw LLM artifacts, and the final review.

OUT_DIR="Papername_out"
mkdir -p "$OUT_DIR"

Write $OUT_DIR/run_config.json before running the pipeline, and update it as resolved metadata or options become known. Include all parameters: target DOI or OpenAlex ID, resolved target title, output directory, stop point, aliases, full-text setting, classification method, LLM execution path, provider/model if external cloud/API is used, review length, references/bibliography setting, paper groups to include, prompt-size limits, PDF limits, smoke-test limits, and the commands or internal-agent steps used.

Use this shape as the minimum config:

{
  "target": "",
  "resolved_title": "",
  "output_dir": "Papername_out",
  "stop_point": "write_review",
  "aliases": [],
  "use_full_text": true,
  "classification_method": "llm",
  "llm_execution": "internal_agent",
  "external_provider": null,
  "external_model": null,
  "review_length_pages": null,
  "include_references_bibliography": true,
  "include_papers": "yes",
  "limits": {},
  "commands_or_steps": []
}

Fetch citing papers:

python3 src/fetch_openalex_citations.py TARGET \
  --output "$OUT_DIR/CitePaper.csv" \
  --max-results 5

If the paper title is only learned after fetching, rename the output directory to the resolved-title Papername_out form, then update run_config.json.

If the selected stop point is fetch papers, stop after CitePaper.csv, CitePaper.meta.json, and run_config.json have been preserved in OUT_DIR.

If using full text, download open PDFs and extract text:

python3 src/download_open_pdfs.py \
  --input "$OUT_DIR/CitePaper.csv" \
  --out-dir "$OUT_DIR/open-pdfs"

python3 src/extract_pdf_text.py \
  --manifest "$OUT_DIR/open-pdfs/manifest.csv" \
  --out-dir "$OUT_DIR/open-pdfs-text"

For the default internal-agent LLM classification path, read $OUT_DIR/CitePaper.csv, $OUT_DIR/CitePaper.meta.json, and extracted text from $OUT_DIR/open-pdfs-text when available. Classify each paper as yes, no, or dont_know, then write $OUT_DIR/classified_papers.csv with the columns expected by src/build_review_prompt.py. Preserve the classification evidence and reasoning in CSV audit fields where possible.

For the regex path, classify with metadata-derived regexes and user aliases:

python3 src/keywords.py \
  --input-csv "$OUT_DIR/CitePaper.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --output "$OUT_DIR/classified_papers.csv" \
  --alias ALIAS

If the selected stop point is classify papers, stop after classified_papers.csv, supporting prompt/evidence files, and run_config.json have been preserved in OUT_DIR.

Build an abstract-only review prompt:

python3 src/build_review_prompt.py \
  --classified "$OUT_DIR/classified_papers.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --output "$OUT_DIR/review_prompt.md"

For full-text review prompts, include the extracted text directory:

python3 src/build_review_prompt.py \
  --classified "$OUT_DIR/classified_papers.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --full-text-dir "$OUT_DIR/open-pdfs-text" \
  --full-text-chars 8000 \
  --output "$OUT_DIR/review_prompt_fulltext.md"

If the user requested prompt generation only, stop after preserving the prompt Markdown file. Otherwise, write $OUT_DIR/review_draft.md using the internal agent by default. Use the prompt file as the review-writing input and respect the requested review length when one was provided. When include_references_bibliography is true, write the review like a scientific paper: cite the target paper and discussed citing papers in the text, then add a References or Bibliography section at the end. Build bibliography entries from CitePaper.csv and CitePaper.meta.json; include title, year, venue/source when available, DOI, and OpenAlex URL. If author metadata is unavailable, use a title-first reference entry rather than inventing authors.

Write a cloud/API LLM review draft only when the user has chosen a provider and model. If include_references_bibliography is true, preserve a prompt file in OUT_DIR that explicitly asks for in-text references and an end bibliography before calling the external provider:

python3 src/llm_write_review.py \
  --provider PROVIDER \
  --model MODEL \
  --mode selective \
  --prompt "$OUT_DIR/review_prompt.md" \
  --output "$OUT_DIR/review_draft.md"

Generated-Regex Route

Use this when aliases and title-derived metadata are not precise enough, but the user wants an auditable regex classifier instead of per-paper LLM calls.

python3 src/prepare_regex_context.py \
  --input-csv "$OUT_DIR/CitePaper.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --output "$OUT_DIR/regex_context.md" \
  --alias ALIAS

For internal-agent regex generation, do not call src/llm_generate_regex.py. Instead, read docs/paper_usage_regex_prompt.md and $OUT_DIR/regex_context.md, then write $OUT_DIR/generated_keywords_regexp.py directly.

For external cloud/API regex generation, use:

python3 src/llm_generate_regex.py \
  --provider PROVIDER \
  --model MODEL \
  --prompt docs/paper_usage_regex_prompt.md \
  --context "$OUT_DIR/regex_context.md" \
  --output "$OUT_DIR/generated_keywords_regexp.py"

Then classify with the generated regex file:

python3 src/keywords.py \
  --input-csv "$OUT_DIR/CitePaper.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --mode generated \
  --regexp "$OUT_DIR/generated_keywords_regexp.py" \
  --output "$OUT_DIR/classified_papers.csv"

Treat generated regex files as trusted local Python. Inspect them before use when they came from any LLM, internal or external. Preserve the instruction prompt, regex_context.md, generated regex file, and classified CSV for auditability.

Direct LLM Classification

Use this when paper-use evidence is subtle. Prefer internal-agent LLM classification when the user has not explicitly chosen an external cloud/API provider. The classifier returns yes, no, or dont_know plus audit fields.

For internal-agent classification, read the citation CSV, target metadata, and abstract/full-text evidence directly, then write the same classified_papers.csv shape used by the scripts. Preserve per-paper reasons, evidence quotes, and confidence where practical.

For external cloud/API classification, use:

python3 src/llm_classify_papers.py \
  --input-csv "$OUT_DIR/CitePaper.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --provider PROVIDER \
  --model MODEL \
  --evidence-mode abstract \
  --output "$OUT_DIR/classified_papers.csv" \
  --raw-dir "$OUT_DIR/llm-classification-raw"

For full-text classification, download and extract open PDFs before classifying:

python3 src/download_open_pdfs.py \
  --input "$OUT_DIR/CitePaper.csv" \
  --out-dir "$OUT_DIR/open-pdfs"

python3 src/extract_pdf_text.py \
  --manifest "$OUT_DIR/open-pdfs/manifest.csv" \
  --out-dir "$OUT_DIR/open-pdfs-text"

python3 src/llm_classify_papers.py \
  --input-csv "$OUT_DIR/CitePaper.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --provider PROVIDER \
  --model MODEL \
  --evidence-mode fulltext \
  --full-text-dir "$OUT_DIR/open-pdfs-text" \
  --full-text-chars 30000 \
  --output "$OUT_DIR/classified_papers.csv"

Full-Text Review Prompt

Full text is optional for review writing. Downloaded PDFs are converted to text, then selected snippets are inserted into the Markdown prompt.

python3 src/build_review_prompt.py \
  --classified "$OUT_DIR/classified_papers.csv" \
  --metadata "$OUT_DIR/CitePaper.meta.json" \
  --full-text-dir "$OUT_DIR/open-pdfs-text" \
  --full-text-chars 8000 \
  --output "$OUT_DIR/review_prompt_fulltext.md"

Use docs/Control-Review-size.README.md when raising --abstract-chars, --full-text-chars, --max-input-chars, or --max-output-tokens.

Orchestrator

Use src/run_pipeline.py for quick one-command runs. It resolves bundled scripts from src/ and the generated-regex prompt from docs/, while output artifacts are written relative to the current working directory unless explicit paths are passed. Always pass explicit paths into OUT_DIR when using the orchestrator or individual scripts.

python3 src/run_pipeline.py TARGET \
  --alias ALIAS \
  --citation-csv "$OUT_DIR/CitePaper.csv" \
  --classified-output "$OUT_DIR/classified_papers.csv" \
  --review-output "$OUT_DIR/review_prompt.md" \
  --pdf-dir "$OUT_DIR/open-pdfs" \
  --fulltext-dir "$OUT_DIR/open-pdfs-text" \
  --fulltext-review-output "$OUT_DIR/review_prompt_fulltext.md" \
  --llm-regex-output "$OUT_DIR/generated_keywords_regexp.py" \
  --llm-review-output "$OUT_DIR/review_draft.md" \
  --max-results 5

For complex or partially manual workflows, use the explicit script commands above.

Review Standards

• Prefer dont_know over a weak yes; false positives damage the review more than uncertain papers that can be manually checked.
• Put every generated file for a run inside one Papername_out folder. Do not scatter CSV files, prompt files, PDFs, extracted text, generated regexes, or final reviews outside that folder.
• Keep $OUT_DIR/run_config.json with all selected/defaulted parameters and update it when resolved metadata, stop point, or output paths change.
• If include_references_bibliography is true, include in-text references and an end bibliography for the target paper and all papers discussed in the review. Do not fabricate missing bibliographic fields.
• Preserve auditability and accountability: keep all generated prompt Markdown files, including regex_context.md, review_prompt.md, review_prompt_fulltext.md, chunk prompts, synthesis prompts, and any raw LLM prompts/responses produced during the run.
• Distinguish abstracts-only evidence from full-text evidence in the final answer or draft.
• Report counts for fetched papers, classified yes, no, and dont_know, downloaded PDFs, extracted texts, and papers included in the prompt.
• Keep classified_papers.csv, generated regexes, and raw LLM classification responses when they were produced.
• Do not claim the pipeline read paywalled or unavailable full texts.
• Before large external cloud/API calls, estimate prompt size and confirm the user is comfortable with the cost and data exposure.

Final Response

When completing a task with this skill, summarize:

• The route used: metadata regex, generated regex, direct LLM, abstract-only, or full-text, and whether LLM work was internal-agent or external cloud/API.
• Stop point reached: fetch papers, classify papers, or write the review.
• The Papername_out output folder path and run_config.json path.
• Key output files written inside the output folder.
• All preserved prompt files generated during the run.
• Classification counts and any notable uncertainty.
• Provider/model used for external cloud/API steps, if any.
• Important limitations, especially missing abstracts, missing full text, or cases that need manual review.
• A message suggesting the user check run_config.json to re-run with adjusted parameters, for example: "To re-run or tune this review, start from $OUT_DIR/run_config.json and change the parameters you want."