Paper discovery

Research Paper Discovery

Purpose

Retrieve scholarly literature across the five package APIs into a provenance-rich inventory.

When To Use

• Building or expanding a seed set of papers for a project.
• Walking forward citations or backward references from a known paper.
• Querying shaped recommendations from positive and negative seeds.
• Searching for recent preprints not yet indexed by major databases.
• Enriching DOIs with venue, license, funding, or retraction status.
• Finding the best legal open-access copy of a paper.

Handoffs

Upstream inputs are the research question, seed sources, search axes, time window, domain filters, license constraints, and commercial-use constraints.

Downstream artifacts are Run Summary, Search Protocol, Result Inventory, Citation Expansion, Recommendations, Licensing And Reuse Notes, Retraction And Hygiene Flags, Gaps And Misses, and Verification Backlog.

Pass the inventory to literature-review for corpus audit and gap analysis. Pass relevant source subsets or citation provenance to report-arbitration, research-manuscript-audit, or update-docs when those skills need primary-source evidence.

Required Inputs

• {{RESEARCH_QUESTION}}: the governing question that scopes the search.
• {{SEARCH_AXES}}: the concepts, sub-questions, or operational dimensions to sweep.
• {{AS_OF_DATE}}: date used for freshness judgments and recency filters.

Optional Inputs

• {{SEED_SOURCES}}: trusted papers, DOIs, or arXiv IDs to anchor expansion.
• {{TIME_WINDOW}}: publication-year range or "last N months" cap.
• {{DOMAIN}}: field, arXiv categories, or topic IDs to filter on.
• {{LICENSE_CONSTRAINTS}}: e.g. "OA only", "CC-BY only", "no S2 data downstream".
• {{COMMERCIAL_USE}}: whether outputs feed a commercial product (changes what licensing you must respect).
• {{OUTPUT_LOCATION}}: run directory, manifest path, or report destination.
• {{API_KEYS}}: which keys are available (S2, OpenAlex). Affects throughput and budget.

Authority Order

Use {{SHARED_RESEARCH_CONTRACT}} when supplied. Otherwise apply this retrieval-specific order:

1. Explicit user instructions, release constraints, and licensing constraints.
2. Primary API records, including paper objects, citation edges, license fields, and retraction flags.
3. Cross-source corroboration: a paper returned by two independent indexes outranks one.
4. Source-of-record sites (publisher page, arXiv abstract page, repository landing page) when the API field is sparse, stale, or ambiguous.
5. Aggregated metadata from a single index, labeled with that index.
6. Inferred or imputed fields, labeled as inference.

If sources disagree (e.g. publication year, author list, OA status), record the disagreement and the competing values.

Tool And File-Access Assumptions

This skill assumes network access to the five APIs below and the ability to write a run directory. If retrieval is unavailable, state the limitation, list the queries that would have been run, and produce a Verification Backlog. Report only retrieved results.

The API behavior described below is the contract. Implement it with a CLI, notebook, agent tool, SDK, or direct curl; the reference invocations are written as shell command shapes for portability.

When To Use Which API

Task	Best API	Why
Keyword search (recent / niche)	arXiv	Indexes preprints within hours. Supports category scoping (cs.CR, cs.AI, cs.CL, cs.LG, cs.SE).
Keyword search (broad / historical)	Semantic Scholar	Covers non-arXiv venues (ACL, USENIX, NeurIPS proceedings, journals). Returns citation counts for ranking.
Keyword search + rich filtering	OpenAlex	250M+ works. Filter by year, citation count, OA status, topic, institution, funder. Also offers first-party full-text PDF and TEI XML download (~60M works).
Citation graph walks	Semantic Scholar	Forward and backward traversal from a known paper ID. Unique capability across these five.
Recommendations from seeds	Semantic Scholar	Shaped recommendation endpoint with positive and negative examples. Unique capability across these five APIs.
ID resolution to S2 metadata	Semantic Scholar	Converts arXiv / DOI to S2 paper IDs for graph operations.
Batch metadata (categories, versions)	arXiv	Version history, primary and secondary categories, author affiliations, DOIs, journal refs. Up to 100 IDs per request.
DOI normalization and enrichment	Crossref	Bibliographic metadata, license info, funding data, Retraction Watch retraction status, structured reference lists. DOI backbone; coverage varies by member and field.
Best legal OA copy by DOI	Unpaywall	Purpose-built. Indexes ~50M OA articles with license info.
Download arXiv PDFs	arXiv direct	Fetch from arxiv.org/pdf/. For non-arXiv OA, use Unpaywall.

Routing Notes

• arXiv catches preprints before Semantic Scholar indexes them, often by days or weeks. Semantic Scholar covers venues outside arXiv. Use both to reduce blind spots.
• OpenAlex is the broadest index. It now carries usage-based API pricing, first-party full-text downloads, and an official CLI for bulk filtered downloads with checkpointing. Reach for it when you need structured filters (institution, funder, topic), broad non-CS coverage, or batch PDF and TEI XML retrieval.
• Crossref is a DOI backbone and enrichment layer. It exposes what members deposit, so completeness varies by publisher and field. Its Retraction Watch integration is critical for corpus hygiene; check retraction status before including a paper in a production pipeline.
• Unpaywall is the specialist for legal OA retrieval. OpenAlex and Semantic Scholar carry OA signals; Unpaywall is the purpose-built lookup for the best legal copy by DOI with license details.

Licensing And Commercial Use Warnings

Respect provider and per-paper licenses.

• Semantic Scholar limits the standard API and data license to internal, non-commercial research and education. Commercial uses require contacting Semantic Scholar for an expanded license.
• arXiv metadata is CC0. E-print PDF redistribution rights vary per paper across arXiv non-exclusive, CC-BY, CC-BY-SA, CC-BY-NC, and other licenses. Check the license field before redistributing PDFs or building derived works from full text.
• OpenAlex: The data snapshot is CC0. The API has usage-based pricing as of 2026 ($1/day free tier with key, $0.01/day without).
• Crossref: Metadata is freely available via the API. The metadata itself carries no restrictive license; downstream content licensing varies by publisher.
• Unpaywall: Data is freely available. It returns the license of each OA copy; use that field to decide downstream full-text rights.

Scope Limitation: Grey Literature

This stack covers scholarly literature (journals, conferences, preprints, books, datasets). The following practically important material lives outside it:

• Technical reports and whitepapers from labs (Anthropic, OpenAI, Google DeepMind, etc.).
• Organization blogs and research posts.
• GitHub repositories, benchmarks, and codebases.
• Standards documents (OWASP, NIST, MITRE ATLAS, ISO).
• Incident reports and practitioner writeups.
• Public red-team reports, system cards, and model cards.

For grey literature, use direct web search, organization feeds, or manual curation. Label a scholarly sweep as scholarly evidence coverage only.

Environment Variables

Wrappers should fall back to environment variables when CLI flags are omitted.

Variable	Used by	Purpose
SEMANTIC_SCHOLAR_API_KEY	All S2 calls	Authenticated rate (1 req/s vs ~25s unauthenticated). Request key.
OPENALEX_API_KEY	OpenAlex calls	$1/day free tier. Get key.
OPENALEX_MAILTO	OpenAlex calls	Email for polite pool (better throughput even without key).
CROSSREF_MAILTO	Crossref calls	Email for polite pool (faster rate limits).
UNPAYWALL_EMAIL	Unpaywall calls	Email required by API policy (no API key).

Core Rules

• Start discovery with OpenAlex + arXiv for breadth and freshness, then expand with Semantic Scholar for graph structure, then enrich with Crossref + Unpaywall. This is the highest-signal order.
• For Semantic Scholar, prefer the bulk search endpoint over relevance search in most cases.
• Start with 3-6 seed papers you already trust before running shaped recommendations.
• Resolve IDs first. S2 citation, reference, and recommendation endpoints all need Semantic Scholar paper IDs.
• Prefer S2 recommendations plus a 1-hop citation walk for concept discovery. Reserve keyword search for wide literal sweeps and pair it with at least one other surface.
• When an arXiv ID lookup fails on S2, retry with the DOI form DOI:10.48550/arXiv.<id> before giving up.
• Keep S2 fields narrow.
• OpenAlex is the fastest API (~10 RPS) and broadest index. Use it for broad sweeps and structured filters.
• Use Crossref for DOI enrichment (venue, license, funding, retraction status) after you have your shortlist.
• Use Unpaywall to find the best legal OA copy for non-arXiv papers when you have DOIs.
• Always check per-paper licensing before redistributing full text.

Standard Workflow

Quick discovery (single session)

1. OpenAlex broad sweep with filters scoped by {{TIME_WINDOW}} and {{DOMAIN}}. Use OA filters when {{LICENSE_CONSTRAINTS}} requires it.
2. arXiv keyword sweep with category filters to catch the freshest preprints not yet in other indexes.
3. S2 bulk keyword sweep for the same queries to catch non-arXiv venues and pick up citation counts.
4. Merge and deduplicate. Papers returned by multiple sources are higher signal.
5. S2 recommendations with your best 3-6 papers as positive seeds for neighborhood expansion.
6. 1-hop citation walk. paper/{id}/citations on the most important papers to find follow-up work, paper/{id}/references to recover canonical priors.
7. Metadata enrichment. arXiv batch metadata for arXiv papers, Crossref lookup for DOIs (include retraction status).
8. OA retrieval for non-arXiv papers. Unpaywall lookup with DOIs to find the best legal OA copy and its license.
9. Download. Fetch arXiv PDFs from arxiv.org/pdf/, non-arXiv OA PDFs via the Unpaywall best_oa_location.url_for_pdf.

Bounded research run

1. Write a run directory with staged JSON outputs (search results, hydrated metadata, recommendations, expanded graph, final shortlist) and a manifest, so an interrupted run resumes without repeating every request.
2. Use one global sequential pacing policy across all endpoints. Keep concurrency conservative, honor Retry-After when present, and back off exponentially on 429, 500, 502, 503, and 504.
3. Deep-read only the primary papers that survive search, recommendations, and 1-hop graph expansion.

Command Shapes

These describe the API call shape. Implement them with whatever runtime you prefer; equivalent CLIs and SDK calls exist in many languages.

arXiv

• Keyword search with category scoping: GET https://export.arxiv.org/api/query?search_query=cat:cs.CR+AND+all:"prompt injection defense"&sortBy=submittedDate&sortOrder=descending&max_results=50
• Batch metadata: GET https://export.arxiv.org/api/query?id_list=2302.12173,2307.15043,2401.05566
• Title-only search: search_query=ti:"PoisonedRAG"

Semantic Scholar

• Bulk search: GET https://api.semanticscholar.org/graph/v1/paper/search/bulk?query=mechanistic+interpretability+steering&fields=title,abstract,year,publicationDate,authors,citationCount,url,openAccessPdf,publicationTypes
• Batch hydrate: POST https://api.semanticscholar.org/graph/v1/paper/batch with { "ids": ["649def34f8be52c8b66281af98ae884c09aef38b", "ARXIV:2106.15928"] }.
• Resolve IDs: GET https://api.semanticscholar.org/graph/v1/paper/ARXIV:2407.08582?fields=externalIds,title.
• Recommendations: POST https://api.semanticscholar.org/recommendations/v1/papers with { "positivePaperIds": ["ArXiv:2407.08582","ArXiv:2310.06824"], "negativePaperIds": ["ArXiv:2406.11717"] }.
• Forward citations: GET https://api.semanticscholar.org/graph/v1/paper/<paperId>/citations.
• Backward references: GET https://api.semanticscholar.org/graph/v1/paper/<paperId>/references (filter by citationCount client-side).
• Author cluster hydrate: POST https://api.semanticscholar.org/graph/v1/author/batch.

OpenAlex

• Full-text search: GET https://api.openalex.org/works?search=prompt+injection+defense&[email protected]
• Search with filters: GET https://api.openalex.org/works?search=RAG+poisoning&filter=publication_year:2024-2026,is_oa:true&mailto=...
• Highly-cited OA: filter=cited_by_count:>50,is_oa:true&per-page=50.
• Sort by date: sort=publication_date:desc.

Crossref

• DOI lookup: GET https://api.crossref.org/works/10.48550/[email protected]
• Search works: GET https://api.crossref.org/works?query=prompt+injection+large+language+models&rows=10&mailto=...
• Search with date filter: query=RAG+security&filter=from-pub-date:2024-01-01&rows=20.
• Retraction check: look for update-to in the response payload.

Unpaywall

• OA lookup: GET https://api.unpaywall.org/v2/10.1145/[email protected]
• Best PDF URL: read best_oa_location.url_for_pdf from the response.

Download

• arXiv PDF: GET https://arxiv.org/pdf/<arxivId>.pdf. Space bursts; heavy concurrent downloads can trigger 10-30 minute IP blocks.
• Non-arXiv OA: download the URL Unpaywall returned in best_oa_location.url_for_pdf.
• OpenAlex full-text: GET https://content.openalex.org/works/<workId>.pdf?api_key=KEY (and .grobid-xml for TEI XML).
• OpenAlex CLI for bulk filtered downloads: openalex download --api-key KEY --filter "publication_year:2024-2026,is_oa:true" --content pdf --output ./pdfs.
• OpenAlex snapshot via AWS Open Data with anonymous access: aws s3 sync s3://openalex ./openalex-snapshot --no-sign-request.

Validate downloaded PDFs by checking for the %PDF magic bytes before treating them as papers.

Query-Shaping Heuristics

arXiv

• Always scope broad queries with --category or the cat: query operator to keep results inside the intended field.
• Useful CS categories include cs.CR for security, plus cs.AI, cs.CL, cs.LG, and cs.SE.
• Prefer title scope (ti:) for known papers, full scope (all:) for conceptual queries.
• Keyword search is literal; multi-word queries are ANDed. Use sortBy=submittedDate to find the newest work.

OpenAlex

• Full-text search covers title, abstract, and indexed fulltext. Broader than arXiv or S2 keyword search.
• Use filter= for structured constraints: publication_year:2024-2026, is_oa:true, cited_by_count:>10, type:article.
• Strongest for filtering by institution, funder, or topic.
• Basic use works without an API key; a mailto email gets you into the polite pool with better throughput.

Crossref

• lookup is for DOIs you already have. Confirm venue, check retraction status, get funding info.
• Retraction checking: Crossref exposes Retraction Watch data via the REST API. For any serious literature pipeline, especially benchmarking or enterprise-facing synthesis, always check before including a paper. Look for update-to.
• search is keyword-based and weaker than S2 or OpenAlex for discovery. Use it to find DOIs for known papers.
• Useful filters: from-pub-date:2024-01-01, type:journal-article, has-abstract:true, has-full-text:true.
• Access tiers are Public with one concurrent request, Polite with mailto and three concurrent requests, and Plus with a paid API key.
• 429 means back off and retry slower. 403 usually signals a manual block; contact Crossref.

Unpaywall

• Requires an email (API policy); no API key.
• Returns the best legal OA copy with license info (cc-by, cc-by-sa, cc-by-nc, and others). Use the license to decide downstream rights.
• Prefer Unpaywall over guessed OA URLs. It resolves across publisher gold OA, green repository copies, and preprints to the best legal version available.
• For arXiv papers, prefer direct arXiv download. Use Unpaywall for non-arXiv papers when you have a DOI and no direct PDF.

Semantic Scholar Recommendation Shapes

Goal	Positive seeds	Negative seeds
Broad neighborhood	4-6 core papers	none
Method intersection	1-3 papers from each subfield	none
Boundary search	papers that define the target construct	papers from adjacent constructs to exclude
Foundational work	1-2 seeds only	none

Too many positive seeds biases the result toward recent papers. If you want older backbone papers, shrink the positive set and walk references instead.

Rate Limits

Source	Unauthenticated	Authenticated / Polite	Notes
arXiv API	~3s between requests	N/A (public endpoint)	Burst PDF downloads trigger IP blocks (10-30 min). Use OAI-PMH for bulk metadata; the search API is for queries.
Semantic Scholar	~25s between requests, 3-5 quick reqs often hit 429	1 req/s cumulative across all endpoints	Commercial use requires an expanded license.
OpenAlex	~10 RPS	Higher with API key	Usage-based pricing ($1/day free tier with key).
Crossref	5 req/s, 1 concurrent	10 req/s polite (3 concurrent), 150 req/s Plus	Set mailto for polite pool.
Unpaywall	100K req/day	Same with email	Requires email; no API key.

For large discovery runs, start with OpenAlex for breadth, then arXiv for freshness, then S2 for graph expansion, then Crossref and Unpaywall for enrichment and OA retrieval.

Output Contract

Produce a run report with these sections, and write a machine-readable manifest if {{OUTPUT_LOCATION}} is provided:

1. Run Summary covering question, search axes, time window, date, and totals.
2. Search Protocol covering per-API queries, filters, fields, screening criteria, and deduplication rule.
3. Result Inventory as a deduplicated paper table with bibliographic metadata, identifiers, citation count, OA status, license, retraction status, and source indexes.
4. Citation Expansion covering walked papers, retrieved edge counts, and surviving edges after filters.
5. Recommendations covering seeds, recommendation shape, top papers, and overlap rationale.
6. Licensing And Reuse Notes covering per-paper license summary, S2 commercial-use flag if relevant, and redistribution constraints.
7. Retraction And Hygiene Flags covering retracted, withdrawn, or update-to-flagged works.
8. Gaps And Misses covering empty queries, skipped indexes, and grey-literature areas outside scholarly API coverage.
9. Verification Backlog covering failed lookups and re-run needs.

Make the inventory the downstream artifact. Leave interpretation for the audit and synthesis skills.

Quality Gate

Before finalizing, verify:

• Every paper in the inventory has at least one canonical ID (S2 paper ID, arXiv ID, DOI, or OpenAlex work ID).
• Source provenance is recorded per paper (which indexes returned it).
• Licensing and retraction fields are populated where the source supports them, or explicitly marked "not checked" when not.
• Browsing, retrieval, and download claims match completed tool actions.
• Rate-limit failures and skipped queries appear in the verification backlog.
• The output is a retrieval inventory. Interpretation belongs in literature-review, report-arbitration, or research-manuscript-audit.