Ingest
Parse a directory of documents (PDF / DOCX / PPTX / HTML) into a queryable Wikify corpus. Use when the user has local files to turn into a corpus, or to finish an arxiv harvest. Wraps `wikify corpus build` and owns parser-backend choice (docling default, lite), the --out convention, partial-failure policy, and post-build health checks.From its SKILL.md
npx -y skills add fgrillo89/wikify --skill ingestAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
2 things to look at
- no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
- 2 stars2 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.
SKILL.md
4.7 KB, as published. Nobody here has run it
ingest
Turn a folder of documents into a queryable Wikify corpus. This is a
thin, decision-light wrapper over wikify corpus build. Its one real
job is choosing the parser backend and confirming the corpus is healthy
before handing it to a build workflow.
Command
wikify corpus build <source> --out data/corpora/<name> \
[--mode additive|sync] \
[--parser default|lite|docling] \
[--workers N] \
[--openalex/--no-openalex] \
[--allow-partial] \
[--parse-timeout SECONDS]
<source> is a directory of documents (PDF / DOCX / PPTX / HTML).
When invoking through uv run for a long build, use
uv run --no-sync wikify corpus build .... Plain uv run re-syncs the
venv on every call; if a wikify.exe from a prior run is still alive it
holds the console script and the sync aborts with os error 32 before the
build starts. --no-sync skips that mutation. If a build was killed, a
stale wikify.exe/python worker may also be holding the GPU or the
.ingest.lock -- the lock is now reclaimed automatically when its owning
pid is dead, but kill orphaned workers before relaunching.
--parse-timeout (default 1800s) caps each file on the GPU
subprocess-batched backend; a wedged parse (a giant scanned book stuck in
OCR) is killed and counted as a parse failure, recoverable under
--allow-partial.
GPU embeddings (opt-in)
The default onnxruntime is the CPU build, so the embedding stage runs on
CPU (~5/s). The GPU variant cannot be a default dependency -- the
onnxruntime-gpu Linux wheel hard-links the CUDA runtime at import and
breaks CPU/CI hosts. On a CUDA box, enable GPU embeddings after uv sync
by reinstalling the GPU variants in place:
uv pip install --reinstall onnxruntime-gpu fastembed-gpu.
They replace the onnxruntime / fastembed modules in place;
embedding._onnx_providers then selects the CUDAExecutionProvider
automatically (cuDNN is preloaded from torch's bundled libs). Needs a CUDA
runtime matching the onnxruntime-gpu major version on the system path.
Embedding throughput rises ~50-100x (jina-v2-small ~5/s CPU -> ~300-560/s
CUDA). A later uv sync reverts to CPU; re-run the reinstall.
Parser-backend choice
default(=docling): Docling for every format. The first run downloads the Granite-Docling-258M formula model (~258 MB) plus the layout / table models; after the cache is warm the median PDF parses in ~10 s on an Ampere GPU. Pick this when equation/table fidelity matters.lite: CI / low-resource backend (pymupdf4llm + python-docx + python-pptx + trafilatura, no models). Use it where the model downloads are unaffordable.
(default and docling select the same backend.)
Mode
--mode additive(default): keep documents already in the corpus and add the new ones. Use for incremental growth.--mode sync: drop corpus documents no longer present under<source>. Use when<source>is the authoritative set.
Citation enrichment / network
--openalex is ON by default: the refresh stage issues network requests
to api.openalex.org to canonicalise bibliography metadata and surface
in-corpus citation matches. Set OPENALEX_EMAIL for the polite-pool rate
limit (10 req/s). Pass --no-openalex to skip that stage and run fully
offline (air-gapped or no-network builds).
Output convention
Always write to --out data/corpora/<name>, never build/<name>.
Partial-failure policy
Default is OFF: any per-file parse failure aborts the whole build with
exit 5. Pass --allow-partial to continue past failures and recover the
successful papers; a later re-run picks up the files that failed.
Post-build health check
wikify corpus check data/corpora/<name>
The corpus dir is a positional argument (not --out); when omitted it
falls back to WIKIFY_CORPUS or the cwd (see
docs/filesystem-state-design.md, "Corpus build/read commands").
Confirm the doc / chunk / embedding / edge counts look right before
declaring the corpus ready.
Rebuild surface
For re-deriving artifacts on an existing corpus without a full rebuild,
corpus rechunk and corpus refresh are available. Mention them; do not
expand on them here.
References
../wikify/subskills/reference/references/cli/grammar.md../wikify/subskills/reference/references/cli/output-contract.md../wikify/subskills/reference/references/cli/exit-codes.md