agentsclimarketplace

Arxiv

Skill fgrillo89/wikify/.claude/skills/arxiv

Acquire arXiv papers on a topic and stage them for `corpus build`. Use when the user wants to download, harvest, fetch, or bulk-collect arXiv papers about a subject, field, or category, or seed a Wikify corpus from arXiv. Composes `arxiv scout` (discover categories), `arxiv identify` (harvest metadata), `arxiv download` (PDFs), then `corpus build`, owning category selection, coverage threshold, resumability, and partial-failure policy.From its SKILL.md

Install
npx -y skills add fgrillo89/wikify --skill arxiv

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

3.9 KB, as published. Nobody here has run it

arxiv

Acquire arXiv papers for a topic and stage them for corpus build. This workflow owns category selection, the scout coverage threshold, the one-staging-dir-per-request rule, partial-failure policy, and the handoff to ingest. It composes the wikify arxiv CLI.

arXiv harvest is set-based (categories), not free-text. The only real decision is which categories cover the topic; scout answers that, and everything after is mechanical and resumable.

Set WIKIFY_CONTACT_EMAIL before harvesting; arXiv requests a contact in the User-Agent for programmatic use.

Inputs

  • topic - natural-language subject (e.g. "machine learning").
  • out - staging directory, one per category set (e.g. data/staging/<topic-slug>).
  • corpus - target corpus directory for the final corpus build.

Workflow (5 phases)

P1 - Scout

Run scout for the topic across 2-3 phrasings and pool the primary-category histograms.

wikify arxiv scout "all:<topic>" --max 200 --format json
wikify arxiv scout "ti:<topic>"  --max 200 --format json

Field prefixes all: / ti: / abs: / cat: scope the search. Collect primary_histogram[].{category,count,setspec} and total_results across the runs. Scout samples the Query API (capped at 30k hits); it is a discovery aid, not an exhaustive lister.

P2 - Decide categories

Keep categories whose pooled share clears a coverage threshold (default: the categories covering ~90% of sampled hits, dropping any below ~5% of the top category). Show the chosen --category set and the rough size (total_results plus per-category counts) to the user and confirm before harvesting a large set. Rows with an empty setspec are unmappable; only harvest those via --set <raw> if the user asks.

P3 - Identify (resumable)

wikify arxiv identify --category <c1> --category <c2> ... --out <out>

identify exhaustively walks the OAI-PMH record set(s), writes one record per paper to <out>/manifest.jsonl, and stores the resume cursor in <out>/harvest_state.json; re-running resumes from that cursor. An unrecognized archive errors with unknown_category - pass the exact setSpec via --set. A state_mismatch error means <out> was harvested for different categories; use a fresh --out. Check progress any time with wikify arxiv status --out <out>.

P4 - Download (resumable, throttle-aware)

wikify arxiv download --out <out>

Defaults to arXiv's PDF-friendly ~4 req/s (tune with --concurrency / --rate) and backs off on 429/503. It exits non-zero (download_incomplete) if any PDF fails; re-run to resume. Pass --allow-partial only when the user accepts a partial corpus. Re-check with wikify arxiv status --out <out>.

P5 - Hand off to ingest

wikify corpus build <out> --out <corpus>

Ingest enumerates the staged PDFs and ignores manifest.jsonl / harvest_state.json.

Stop conditions

  • Stop after P2 if the user rejects the proposed category set.
  • Stop after P4 if downloads are incomplete and --allow-partial was not granted; report the failed list and how to resume.
  • The manifest is the durable record; every phase is re-entrant on the same --out, so an interrupted run continues where it stopped.

References

  • ../wikify/subskills/reference/references/cli/grammar.md
  • ../wikify/subskills/reference/references/cli/output-contract.md
  • ../wikify/subskills/reference/references/cli/exit-codes.md

Keep looking

Skills are one crate of 609,689. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.