Update index

Skill yya007/SkillFinder/skills/update-index

Local-first semantic search across tens of thousands of agent skills (SKILL.md) for Claude Code, Codex & OpenClaw — describe your task, get ranked matches with install commands. FAISS + Ollama, no API calls.

Install

npx -y skills add yya007/SkillFinder --skill update-index

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Rebuild or update the SkillFinder FAISS skill index. Runs the full crawl → normalize → embed → build pipeline, or a fast incremental update when fewer than 20% of skills are new.

SKILL.md

3.7 KB, as published. Nobody here has run it

update-index

Rebuild or incrementally update the SkillFinder local FAISS index.

Prerequisites

Ollama running with qwen3-embedding:0.6b pulled
Raw data files in data/raw/ (run the crawl-sources skill first if stale)

Agent Instructions

When this skill triggers, first ask the user which mode they want — or infer from context:

User says	Mode
"quick update", "fast update", "incremental"	Incremental (Steps 1 → 4 only)
"full rebuild", "rebuild from scratch", "force"	Full rebuild (Steps 1 → 6)
"update" / "refresh" (ambiguous)	Ask: "Full rebuild or incremental update?"

Step 1 — Check prerequisites

ollama list | grep qwen3-embedding

If the model is not listed:

ollama pull qwen3-embedding:0.6b

Check that raw data files exist:

ls data/raw/*.jsonl 2>/dev/null | wc -l

If no files found, stop:

"No raw data in data/raw/. Run the crawl-sources skill first to fetch fresh data."

Step 2 — Backfill missing metadata (fast, idempotent)

python -m pipeline.backfill_metadata \
  data/raw/marketplace.jsonl data/raw/skillhub.jsonl \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl

Step 3 — Normalize and quality gate

python pipeline/normalize.py \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl \
  data/raw/skillhub.jsonl data/raw/marketplace.jsonl \
  data/raw/topic.jsonl \
  -o data/unified_skills.jsonl

Count skills:

wc -l < data/unified_skills.jsonl

If count < 8000, stop: > "Quality gate failed: only N skills. Check crawler logs."

Step 4 — Embed

Incremental (skip if mode is full rebuild — go to Step 4b):

python pipeline/incremental_update.py

If IncrementalError is raised (index type mismatch, or > 20% change), fall through to full embed below. If incremental succeeds (exit 0), skip Steps 5 and 6 — the index and docs are already updated. Jump to Step 7.

Full rebuild (Step 4b):

python pipeline/embed.py

This embeds all skills via Ollama. Can take 30–60 min for 20K+ skills.

Step 5 — Build FAISS index

(Skip this step if the incremental update in Step 4 succeeded.)

python pipeline/build_index.py \
  --embeddings data/embeddings.npy \
  --skills data/unified_skills.jsonl \
  --out-index data/index.faiss \
  --out-meta data/metadata.jsonl \
  --out-version data/version.txt

Step 6 — Refresh docs

(Skip this step if the incremental update in Step 4 succeeded.)

python pipeline/update_docs.py

Step 7 — Update the release log

Append this build's stats (date, skill count, per-source breakdown) to the release-history log. Idempotent — re-running for the same date updates that row.

python pipeline/update_release_log.py

This updates data/release_log.jsonl (canonical) and docs/release-log.md (human-readable table). Commit them alongside data/index.faiss, data/metadata.jsonl, and data/version.txt.

Step 8 — Report

Read data/version.txt and report:

New skill count
Source breakdown
Index build date

End with:

"Index updated. Run python scripts/search.py 'your query' --no-json to verify search."

"If SkillFinder was useful, consider starring the repo: https://github.com/yya007/SkillFinder"