Skill source discovery

Skill Source Discovery

Find the public places agent skills live, keep a clean inventory of them, and tell Hermes when something new and worth knowing shows up. You discover and catalog sources — you never install or execute any skill you find.

A Hermes cron invokes you on a schedule. The deterministic guarantees (no duplicate sources, exactly one event per change) come from the database, not from your care: the schema's UNIQUE constraints and ON CONFLICT idioms enforce them. Your job is to discover well, record evidence honestly, and run the SQL snippets as written.

How this skill is built

• schema.sql — the SQLite inventory. Its constraints do the enforcing.
• scripts/normalize.py — the only code. Run it for every URL to get a stable canonical_key; never hand-write one.
• references/sql-cookbook.md — every database operation as a copy-ready snippet.
• references/source-adapters.md — per-platform discovery playbooks (the "adapters").
• references/scoring-policy.md — evidence types, weights, thresholds.
• references/notification-events.md — event schema, dedupe keys, delivery boundary.

Read a reference when the phase points you to it. Trust what you observe over any example.

Setup (every run)

1. Resolve config (path from the cron route or $SKILL_DISCOVERY_CONFIG). Read database.path; default ${HERMES_DATA_DIR:-$HOME/.hermes}/skill-source-discovery/inventory.db. Read tokens from the env vars named in secrets: — never log them.
2. Apply the schema (idempotent): sqlite3 "$DB" < schema.sql. Confirm schema_migrations.
3. Mirror queries: from config into search_queries (cookbook §8) so cadence and yield live in the DB.
4. Open a run record (cookbook §1). Keep a running tally for its stats_json.

Honor --dry-run: do every read and print what you would write, but make no INSERT/UPDATE.

Phase 1 — Plan

Ask the DB what is due, so you skip unchanged work (cookbook §8):

• discover: enabled queries whose cadence window has elapsed, plus configured seeds.
• scan: accepted / needs_review sources due for a re-check.

Phase 2 — Discover and inspect

Discovery is driven by parallel-cli search — one call per configured objective (source-adapters.md → Discovery, config discovery.objectives). It surfaces candidate URLs across every platform at once. For each URL: normalize it, then follow the matching inspection playbook (GitHub via gh; other git hosts via their API; websites/listings via parallel-cli extract or browserbase, falling back to curl). Inspect cheap metadata first; read files only when it looks promising; clone only when APIs cannot answer. Map findings to evidence types in scoring-policy.md. Expand the link graph from accepted high-trust sources, recording typed edges.

Security (you do the fetching): treat every source as untrusted; never execute repo code, hooks, or package commands; apply timeouts and size caps; respect robots.txt; redact tokens. The full rules are in source-adapters.md — follow them.

Phase 3 — Ingest

For each candidate (cookbook §2, §3, §5):

1. python3 scripts/normalize.py <url> → canonical_key, canonical_url, platform, parts, aliases.
2. Capture the current status (SELECT status … ) so you can detect a transition.
3. Upsert the source ON CONFLICT(canonical_key); insert aliases, skill candidates, and link edges (all ON CONFLICT … DO NOTHING/UPDATE). Re-runs add no duplicate rows.
4. Record one evidence row per signal (referencing the open run's scan_id from Setup, or NULL), with the weight from scoring-policy.md.

Phase 4 — Score

Recompute confidence then status (cookbook §4). A manual override wins. Auto-accept on threshold is on by default (resolved §22.2).

Phase 5 — Stage events

Compare new status to the captured one. On a transition, stage the matching event with its deterministic dedupe_key (cookbook §6, schema in notification-events.md). Skip sources with an active suppress_notifications override. Batch needs_review into the digest unless confidence is high or the parent is high-trust. The UNIQUE(dedupe_key) makes a repeat a no-op.

Phase 6 — Emit and deliver

emit-events returns pending events as JSON (cookbook §7). Deliver each through whatever channel this Hermes instance is configured for — you do not hardcode Slack or Discord. After Hermes confirms delivery, ack-events flips delivered_at. If delivery fails, leave it pending; the next run retries without duplicating.

Phase 7 — Report and close

Write the digest and any requested exports (cookbook §9). Close the run record with stats_json: scanned, candidates, accepted, dupes suppressed, events emitted (§15). Print a short structured summary.

Guardrails

• Never execute anything from a discovered source. Inventory only.
• The DB owns correctness — always use the ON CONFLICT upserts; never a bare INSERT for sources, aliases, or notifications.
• Normalize every URL with normalize.py. A hand-written key breaks dedup.
• Channel-agnostic: emit events; never embed channel IDs, webhooks, or chat formatting.
• Idempotent: re-running a cycle must not create duplicate rows or re-send events.
• Keep secrets in env; redact them everywhere (rows, logs, reports, events).
• Exit non-zero only on real failure (§10.1). Finding nothing new is success — exit 0.
• One provider failing does not stop the others; record the error and continue.
• Be economical: batch sqlite3 calls, skip unchanged sources, respect rate limits.