agentsclimarketplace

Ingest

Skill fgrillo89/wikify/.claude/skills/ingest

Parse a directory of documents (PDF / DOCX / PPTX / HTML) into a queryable Wikify corpus. Use when the user has local files to turn into a corpus, or to finish an arxiv harvest. Wraps `wikify corpus build` and owns parser-backend choice (docling default, lite), the --out convention, partial-failure policy, and post-build health checks.From its SKILL.md

Install
npx -y skills add fgrillo89/wikify --skill ingest

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

4.7 KB, as published. Nobody here has run it

ingest

Turn a folder of documents into a queryable Wikify corpus. This is a thin, decision-light wrapper over wikify corpus build. Its one real job is choosing the parser backend and confirming the corpus is healthy before handing it to a build workflow.

Command

wikify corpus build <source> --out data/corpora/<name> \
  [--mode additive|sync] \
  [--parser default|lite|docling] \
  [--workers N] \
  [--openalex/--no-openalex] \
  [--allow-partial] \
  [--parse-timeout SECONDS]

<source> is a directory of documents (PDF / DOCX / PPTX / HTML).

When invoking through uv run for a long build, use uv run --no-sync wikify corpus build .... Plain uv run re-syncs the venv on every call; if a wikify.exe from a prior run is still alive it holds the console script and the sync aborts with os error 32 before the build starts. --no-sync skips that mutation. If a build was killed, a stale wikify.exe/python worker may also be holding the GPU or the .ingest.lock -- the lock is now reclaimed automatically when its owning pid is dead, but kill orphaned workers before relaunching.

--parse-timeout (default 1800s) caps each file on the GPU subprocess-batched backend; a wedged parse (a giant scanned book stuck in OCR) is killed and counted as a parse failure, recoverable under --allow-partial.

GPU embeddings (opt-in)

The default onnxruntime is the CPU build, so the embedding stage runs on CPU (~5/s). The GPU variant cannot be a default dependency -- the onnxruntime-gpu Linux wheel hard-links the CUDA runtime at import and breaks CPU/CI hosts. On a CUDA box, enable GPU embeddings after uv sync by reinstalling the GPU variants in place: uv pip install --reinstall onnxruntime-gpu fastembed-gpu.

They replace the onnxruntime / fastembed modules in place; embedding._onnx_providers then selects the CUDAExecutionProvider automatically (cuDNN is preloaded from torch's bundled libs). Needs a CUDA runtime matching the onnxruntime-gpu major version on the system path. Embedding throughput rises ~50-100x (jina-v2-small ~5/s CPU -> ~300-560/s CUDA). A later uv sync reverts to CPU; re-run the reinstall.

Parser-backend choice

  • default (= docling): Docling for every format. The first run downloads the Granite-Docling-258M formula model (~258 MB) plus the layout / table models; after the cache is warm the median PDF parses in ~10 s on an Ampere GPU. Pick this when equation/table fidelity matters.
  • lite: CI / low-resource backend (pymupdf4llm + python-docx + python-pptx + trafilatura, no models). Use it where the model downloads are unaffordable.

(default and docling select the same backend.)

Mode

  • --mode additive (default): keep documents already in the corpus and add the new ones. Use for incremental growth.
  • --mode sync: drop corpus documents no longer present under <source>. Use when <source> is the authoritative set.

Citation enrichment / network

--openalex is ON by default: the refresh stage issues network requests to api.openalex.org to canonicalise bibliography metadata and surface in-corpus citation matches. Set OPENALEX_EMAIL for the polite-pool rate limit (10 req/s). Pass --no-openalex to skip that stage and run fully offline (air-gapped or no-network builds).

Output convention

Always write to --out data/corpora/<name>, never build/<name>.

Partial-failure policy

Default is OFF: any per-file parse failure aborts the whole build with exit 5. Pass --allow-partial to continue past failures and recover the successful papers; a later re-run picks up the files that failed.

Post-build health check

wikify corpus check data/corpora/<name>

The corpus dir is a positional argument (not --out); when omitted it falls back to WIKIFY_CORPUS or the cwd (see docs/filesystem-state-design.md, "Corpus build/read commands"). Confirm the doc / chunk / embedding / edge counts look right before declaring the corpus ready.

Rebuild surface

For re-deriving artifacts on an existing corpus without a full rebuild, corpus rechunk and corpus refresh are available. Mention them; do not expand on them here.

References

  • ../wikify/subskills/reference/references/cli/grammar.md
  • ../wikify/subskills/reference/references/cli/output-contract.md
  • ../wikify/subskills/reference/references/cli/exit-codes.md

Keep looking

Skills are one crate of 609,689. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.