Weave

Weave

Weave maintains a private, provenance-backed social graph of people, relationships, preferences, and shared experiences — queryable for meeting prep, gift ideas, hosting, introductions, city connections, and serendipity discovery. Every stored fact carries source type, reference, timestamp, and confidence score; the graph never silently merges two person records and never writes back to external systems without explicit per-sync approval.

When to Use

• Contact management and relationship tracking
• Social graph queries (who knows whom, how)
• Contact enrichment from multiple sources
• Store or update information about a person, relationship, or preference
• Prepare for a meeting, dinner, or introduction
• Find connections in a given city
• Generate gift ideas grounded in known preferences
• Discover serendipity connections between people
• Sync contacts from Google Contacts or Clay

When NOT to Use

• Sending messages or emails (use Dispatch)
• Calendar management (use Sands)
• OSINT research (use Scout)
• Knowledge graph entity resolution
• Web research without a social graph need — use Sift
• CRM or sales pipeline automation
• Personality profiling without evidence

Auth Rule — <operator> Only

Weave exclusively uses <operator>'s Google auth (<user-google-email>). Never use the agent's account for any Weave operation. The TOKEN_PATH in google_sync.py is hardcoded to <user-google-email>.json. Violation silently fetches wrong contact data.

ALWAYS sync Contacts after changes via scripts/google_sync.py. This is the canonical sync path — never skip it after a contact mutation.

Responsibility Boundary

Weave owns the social relationship graph: people, relationships, preferences, and shared experiences. It is the only skill that writes to its SQLite database (weave.sqlite).

Backend: SQLite with WAL mode via weave_sqlite.WeaveDB. Replaced LadybugDB in June 2026. See references/sqlite-backend-research.md for the evaluation and migration details.

Weave does not: perform OSINT research (Scout), manage calendars (Sands), or organize files (Bower).

Ontology types

• Entity/Person — people in the social graph. Weave extracts and manages Person entities exclusively.

Weave may optionally include Chronicle signals in journal payloads for Person nodes with high-confidence identity markers.

SQLite usage guide

Read references/schemas.md for the Python usage pattern and schema details. Read references/query_patterns.md for all SQL query templates. Import: from weave_sqlite import WeaveDB

Storage layout

See references/schemas.md for the storage layout and record schemas. See references/config-defaults.md for default config.

Database rules

SQLite with WAL mode. Multiple concurrent readers + single writer. No lock contention — any process can read while another writes. If a write fails, surface the error, do not retry silently.

Auto-initialization

Every command that opens the database runs _ensure_init() first via WeaveDB.__init__(). No manual init command needed.

Commands

• weave.upsert.person — Add or update a person. Auto-inits DB on first call.
• weave.upsert.relationship — Add or update a Knows edge. Confirm both Person nodes exist first.
• weave.upsert.preference — Store a provenance-backed preference.
• weave.import.csv — Bulk import contacts via COPY FROM. Read references/import_export.md.
• weave.query — Query the graph. Modes: lookup, connection, serendipity, city, summarize, gift. Return only stored facts with provenance.
• weave.attach — Query an external skill database read-only.
• weave.export — Export data via COPY TO.
• weave.sync.google-contacts — Bidirectional Google Contacts sync. Read references/connectors.md.
• weave.sync.clay — Bidirectional sync with Clay.
• weave.project.vcard — Generate vCard 4.0 draft.
• weave.writeback.contacts — Push records to Google Contacts or Clay. Disabled by default.
• weave.init — Diagnostic and repair.
• weave.status — Report graph health and config state.
• weave.journal — Write journal for the current run.
• weave.update — Pull latest skill package from GitHub.

Workflow

The Weave social graph pipeline: record → enrich → query → discover.

• Record people, relationships, preferences, and shared experiences
• Enrich contacts via Scout/Sift/Sherlock pipeline
• Query the graph for recall, gifting, hosting, introductions
• Discover serendipitous connections between people

Run completion

After every Weave command:

• Persist any new or updated records to the database.
• Log material decisions to decisions.jsonl.
• Write journal via weave.journal.
• Read-back verification: After every write, immediately query the DB by primary key. Confirm written data matches intent. Never claim success unconfirmed.

Provenance

Every written fact requires: source_type (direct / inferred / imported / user-stated), source_ref, record_time (ISO 8601), confidence (0.0–1.0).

Enrichment Pipeline Execution

Pre-Enrichment Checklist

1. Run inbound Google sync
2. Check SearXNG health 2.5. Probe ALL discovery sources before committing to the run: python3 scripts/discovery_probe.py. If every source reports down (web_search empty, SearXNG engines suspended, DDG anomaly-blocked, no LinkedIn MCP), do NOT start per-contact processing — follow the Discovery Source Availability & No-Fabrication Rule (defer real people, skip non-persons/unresolvables, write no facts). If ≥1 source is live, proceed with the fallback chain in references/discovery-fallback.md.
3. Clear pre-existing garbage — scan for known junk values before enriching so COALESCE preserves nothing
4. placeholder — the parents[2] path bug in scripts can create stale DB files at <hermes-home>/commons/db/ocas-weave/weave.sqlite, <hermes-home>/profiles/commons/db/ocas-weave/weave.sqlite, and <hermes-home>/profiles/indigo/skills/commons/db/ocas-weave/weave.sqlite. Only the canonical path (<hermes-home>/profiles/indigo/commons/db/ocas-weave/weave.sqlite) is correct. Stale DBs confuse subagent enrichment writes. Remove them before enriching.
5. Check edges FK constraint — run python3 -c "import sqlite3; c=sqlite3.connect('<hermes-home>/profiles/indigo/commons/db/ocas-weave/weave.sqlite'); r=c.execute('PRAGMA foreign_key_list(edges)').fetchall(); print(r)". If target_id references persons(id), run python3 scripts/migrate_edges_fk.py before enriching. The target_id column is polymorphic (can point to facts.id or preferences.id) — a wrong FK causes HasFact edge inserts to fail silently.
6. Query contacts with gaps
7. Process each contact through Scout → Sift → Sherlock → Write
8. Run periodic Google sync after every 10 enriched contacts
9. Final Google sync

Unresolvable Contacts Protocol

Read references/unresolvable-contacts.md for the full decision flow. Key rule: when a contact's name is common and no email/phone/location disambiguates, skip — never guess a match. Log to decisions.jsonl with reason skip_unresolvable.

Script note: Shared enrichment extraction, search, and validation logic lives in scripts/weave_enrich.py. Both quick_enrich.py and overnight_enrichment.py import from it. To run overnight enrichment: python3 overnight_enrichment.py. To get contacts with gaps, query the Weave DB directly (see references/query_patterns.md). To check SearXNG health, use the diagnostic curl in enrichment-pipeline.md.

For manual enrichment of high-value contacts, use the full quality pipeline:

1. Pre-Search Seed Quality Check — Check source_type and confidence.
2. Read — Query Weave for the existing Person record.
3. Search — Use SearXNG for identity-resolved research.
4. Enrich (Scout → Sift → Sherlock) — Full pipeline.
5. Write — MERGE on Person, CREATE on Fact/Preference. Always read back.
6. Search again — Follow-up with enriched data.
7. Sync — After writes touching mapped fields, sync to Google Contacts.

Google Contacts sync

See references/connectors.md for full sync rules. Key points:

• Match by google_resource_name, then email, then phone. Never match on name alone.
• Gap-fill only — Weave provenance wins conflicts.
• Outbound requires writeback.google_contacts: true AND a previous sync checkpoint.

Recovery behavior

See references/recovery-weave.md for the full recovery contract.

Discovery Source Availability & No-Fabrication Rule

Every enrichment run depends on a working Scout source. In agent/cron context these sources are external and can fail as a group. There is no point in the pipeline where fabricating occupation/org data is acceptable. If you cannot discover real data, you MUST NOT write it.

At pipeline start, run python3 scripts/discovery_probe.py to test which sources are live. Interpreting results:

• web_search (MCP) — verify in-session it returns non-empty data.web. Empty success:true payloads = non-functional; treat as down.
• SearXNG — see unresponsive_engines. brave: too many requests = rate-limited, recovers after backoff (space queries 15–20s apart; retry with 2**(n+1)*5s sleeps). karmasearch: access denied = needs re-grant, won't recover this run. A single 0-result response is NOT proof of death — a later query can return results.
• DuckDuckGo HTML — last-resort discovery (curl + regex on result__a); hard rate-limits to an HTTP 202 anomaly page after a few queries. Never burn DDG on test queries. See references/discovery-fallback.md.

Decision matrix when Scout is degraded/unavailable:

Situation	Action
≥1 source live	Proceed with fallback chain (LinkedIn MCP → web_search → SearXNG → DDG).
ALL sources down	Do not run Sift/Sherlock on zero input. Instead: (a) skip non-person/business + unresolvable contacts normally; (b) defer real people who already carry recoverable Google-sync context (missing only one field) — they are queued, not dropped; (c) log a pipeline_blocked decision with "stage":"scout" and the unavailable-source list; (d) write NO enrichment facts.
Some contacts resolvable, others not	Enrich the resolvable ones; defer/skip the rest per references/unresolvable-contacts.md.

No-fabrication is non-negotiable: an empty discovery result means "no data," never "write a best guess." Reporting a blocked run honestly (and deferring real people) is the correct outcome. See references/discovery-fallback.md for the full fallback playbook and scripts/discovery_probe.py.

Constraints

See references/constraints.md for the full constraint set.

Gotchas

See references/gotchas-weave.md for the full gotcha catalog including:

• SQLite WAL mode and concurrent access
• Google OAuth token handling and cross-account contamination
• LinkedIn profile fetching
• Python environment (no liblbug.so needed)
• Cron mode constraints and workarounds: execute_code is BLOCKED in cron jobs — it runs arbitrary local Python including subprocess calls that bypass approval. Use terminal with inline python3 -c "..." or python3 /path/to/script.py for all Python operations. Set timeout appropriately (max 600s for foreground). Background processes with notify_on_complete=true work for long sync jobs.
• Contact merge diagnosis and repair

OKRs

Read references/okrs.md for Weave-specific OKR definitions and targets.

Optional skill cooperation

• Chronicle — read for entity enrichment; entity observations emitted via journal payloads
• Scout — receive OSINT findings about people as upsert candidates
• Dispatch — provide social graph context for communication drafting
• Clay (Mesh MCP) — CRM sync via Smithery

Journal outputs

• Observation Journal — query runs, upsert runs, import runs
• Action Journal — sync runs, writeback runs

Initialization

On first invocation, _open_db() handles auto-initialization. See references/init_pattern.md.

Background tasks

Job name	Schedule	Command
weave:update	0 0 * * *	weave.update
weave:sync-google	0 4 * * *	AGENT_ROOT=<hermes-home>/profiles/indigo HOME=/root python3 -u {skill_root}/scripts/google_sync.py
weave:enrichability-recalc	0 1 * * *	python3 {skill_root}/scripts/recalculate_enrichability.py

⚠️ CRON INVOCATION: IGNORE THE RUNBOOK IN THE MESSAGE

The cron job's user message often contains a hardcoded pipeline runbook that is STALE. It may reference removed components (enrichment_data.py, LadybugDB bridge, systemctl stop ladybug-bridge-weave.service). ALWAYS defer to this skill's own documentation over the runbook in the invocation message. The skill is updated first; the cron message template lags by weeks or months.

Specific runbook instructions to IGNORE:

• "Stop the LadybugDB bridge" → NOOP (bridge removed June 2026)
• "Run enrichment_data.py" → DOES NOT EXIST (use WeaveDB queries directly)
• "Run google_sync.py without AGENT_ROOT" → WILL FAIL (must set AGENT_ROOT=<hermes-home>/profiles/indigo HOME=/root)
• "Restart bridge after" → NOOP (bridge removed)
• Any step using execute_code → BLOCKED IN CRON (use terminal + temp file instead)

Agent-Driven Overnight Enrichment

When the enrichment pipeline is run as a cron job (agent-driven, not script-driven), the agent has access to all MCP tools including web_search, web_extract, Composio LinkedIn, and SearXNG. In this mode:

Script Dependencies

• enrichment_data.py does NOT exist on disk. Do not attempt to run it. Instead:
  • For SearXNG health: curl -s "http://localhost:8888/search?q=test&format=json&limit=3"
  • For contacts with gaps: query WeaveDB directly (SELECT p.id, p.name, ... FROM persons p LEFT JOIN edges e ... HAVING occupation IS NULL OR org IS NULL)
  • For writing enrichment: use WeaveDB.execute_write() directly — see Enrichment Write Pattern below
  • For stats: query WeaveDB directly

Enrichment Write Pattern (Agent-Driven)

The WeaveDB schema requires three operations per contact. The facts table has NO person_id column — linkage is via the edges table.

Schema reference:

• persons: id, name, email, phone, location_city, location_country, occupation, org, google_resource_name, clay_id, source_type, source_ref, confidence, record_time, valid_from, valid_until
• facts: id, predicate, value, confidence, source_type, source_ref, record_time (NO person_id)
• edges: id, source_id, target_id, rel_type, strength, since, context, source_ref, confidence, record_time

Write pattern (use terminal with inline Python — execute_code is BLOCKED in cron mode):

import sys, json, uuid
sys.path.insert(0, '<hermes-home>/profiles/indigo/skills/ocas-weave/scripts')
from weave_sqlite import WeaveDB
from datetime import datetime, timezone

db = WeaveDB()
now = datetime.now(timezone.utc).isoformat()

# For each contact:
# 0. READ CURRENT STATE first (only update NULL/empty fields)
current = db.execute("SELECT id, name, occupation, org FROM persons WHERE id = ?", (person_id,))
if not current: skip
person = current[0]

# 1. Build UPDATE dynamically — only set fields that are NULL or empty string
update_fields, update_vals = [], []
if occupation and not person.get('occupation'):
    update_fields.append("occupation = ?"); update_vals.append(occupation)
if org and not person.get('org'):
    update_fields.append("org = ?"); update_vals.append(org)
if not update_fields: skip  # nothing to update

db.execute_write(
    f"UPDATE persons SET {', '.join(update_fields)} WHERE id = ?",
    tuple(update_vals + [person_id])
)
# 2. INSERT fact (the enrichment payload)
fact_id = str(uuid.uuid4())
db.execute_write(
    'INSERT INTO facts (id, predicate, value, source_type, source_ref, confidence, record_time) VALUES (?, ?, ?, ?, ?, ?, ?)',
    (fact_id, json.dumps({"occupation": occupation, "org": org, "confidence": confidence}), source_type, source_ref, confidence, now)
)
# 3. INSERT edge linking person → fact
edge_id = str(uuid.uuid4())
db.execute_write(
    'INSERT INTO edges (id, source_id, target_id, rel_type, source_ref, confidence, record_time) VALUES (?, ?, ?, ?, ?, ?, ?)',
    (edge_id, person_id, fact_id, 'HasFact', source_ref, confidence, now)
)
# 4. READ-BACK VERIFY
verify = db.execute("SELECT occupation, org FROM persons WHERE id = ?", (person_id,))
assert verify[0]['occupation'] == occupation or verify[0]['org'] == org, "WRITE FAILED"

Why not COALESCE? The skill's earlier pattern used COALESCE(?, occupation) which works but makes it impossible to detect "nothing changed" — you always write a fact row even if the data was identical. The explicit state-check pattern above avoids redundant fact writes and makes debugging easier.

Important: Skip contacts where both occupation AND org are null AND confidence < 0.5 — no meaningful data to write.

LadybugDB Bridge

• The ladybug-bridge-weave.service no longer exists after the SQLite migration (June 2026). Do not attempt to stop/start it. The SQLite backend does not require it.

Google OAuth Failure Handling

• If google_sync.py fails with HTTP 401 / invalid_grant, the refresh token has been revoked. The script now exits with code 2 and a clean ABORT message (no traceback). Log the failure and continue with enrichment using MCP tools. Do not halt the entire pipeline. <operator> must re-authorize OAuth manually — there is no programmatic workaround.
• When Google sync is unavailable, enrichment data can still be gathered via web_search, web_extract, Composio web tools, and direct page fetching (curl + Jina Reader).

Page Fetching

• web_extract fails with SearXNG backend ("search-only backend cannot extract URL content"). Use curl -s "https://r.jina.ai/URL" for page content fetching instead.
• LinkedIn profiles: direct HTTP with browser User-Agent works; Jina Reader is blocked for LinkedIn. Composio LINKEDIN_GET_PERSON requires a person_id (not username) — there is no name-search tool.

Self-update

weave.update pulls the latest package from GitHub. See references/self-update.md.

Database maintenance

See references/database_maintenance.md. See references/graph-storage-backend-research.md for the full evaluation of alternatives to LadybugDB (SQLite adjacency lists recommended).

Pitfalls

• Never leave broken scripts after a migration: When you migrate a shared backend (like LadybugDB → SQLite), you MUST update ALL scripts that depend on it in the same session. Do not wait to be told. Check grep -rl "old_import" scripts/ and fix every hit.

• Fix known issues immediately without asking: When you identify a problem and know how to fix it, apply the fix immediately. Do not ask "should I fix this?" or wait for the user to tell you. If something is broken and the fix is clear, just fix it.

• Do not ask confirmation on approved plans: When the user says "yes" to a plan, execute immediately. Do not re-ask "should I proceed?" or present alternatives after approval.

• Module-level imports: Python names imported inside if __name__ == "__main__" are NOT visible to module-level functions. Import timedelta, sqlite3, and all other dependencies at the top of the file.

• Shared auth module: All Google OAuth + API call logic lives in scripts/google_api.py. Import from there — never duplicate get_access_token, api_get, api_post, or api_patch in individual scripts. The shared module handles token refresh, rate-limit backoff, and error handling consistently.

• SQLite FK constraints on polymorphic references: The edges.target_id is a polymorphic reference (can point to persons.id, facts.id, or preferences.id depending on rel_type). Do NOT add FOREIGN KEY (target_id) REFERENCES persons(id) — it breaks HasFact and HasPreference edges. If the FK already exists in a live DB, use a migration script to recreate the table without it (see scripts/migrate_edges_fk.py for the pattern).

• Schema code vs live DB divergence: CREATE TABLE IF NOT EXISTS in weave_sqlite.py only runs on first DB creation. Schema fixes in code do NOT apply to existing DBs. Always write an explicit migration script when changing DDL on a live database, and verify row counts before/after.

• Script name references: When referencing other scripts in subprocess calls or Popen, verify the filename exactly matches what's on disk. enrichment_control.py referenced overnight_weave_enrichment.py but the actual file is overnight_enrichment.py — always ls scripts/ to confirm.

• Enrichment write pattern consistency: When writing enrichment data to Weave, always use weave.execute_write() / weave.execute() (the WeaveDB abstraction layer) rather than raw sqlite3 connections. Raw connections bypass FK enforcement, skip WAL mode, and can leave the DB in an inconsistent state. The only exception is bulk import via weave.bulk_import() which manages its own connection lifecycle.

• Wrong person_id → silent 0-row UPDATE + edge FK failure + orphaned fact (REAL FAILURE MODE): In the three-step write, if the edges INSERT fails with FOREIGN KEY constraint failed but the preceding persons UPDATE raised no error, the cause is almost always a wrong person_id (e.g. a transposed UUID segment — a1e3 vs 1ae3 copied from the gap-query output), NOT DB corruption. The UPDATE matched 0 rows (sqlite3 does NOT error on a 0-row UPDATE), the facts INSERT committed (no FK on facts), and only the edge's source_id → persons(id) FK caught it — leaving an orphaned fact with no edge. Fix/avoid: (1) copy IDs programmatically from the gap-query result, never by hand; (2) before the write loop, assert SELECT id FROM persons WHERE id = ? returns the row; (3) on FK failure, DELETE FROM facts WHERE id = ? the orphan, correct the ID, re-run the full three-step write, and confirm via read-back. Detected and recovered this way on the 2026-07-07 run (Gwendolyn McGinn).

• Printed None from a query = SQL NULL, not the string 'None': When inspecting db.execute() results in terminal output, a field shown as None is Python's None (i.e. SQL NULL), not the literal text "None". A cleanup pass that matches WHERE org = 'None' matches nothing and wastes a cycle. Guard with IS NOT NULL and only treat a value as a string when isinstance(v, str). Several contacts in <operator>'s graph displayed as None in output but were already NULL.

• Enrichment three-step write pattern: Writing enrichment data requires THREE operations because the facts table has no person_id column. The linkage is via edges: (1) UPDATE persons SET occupation=..., org=..., location_city=? WHERE id=?, (2) INSERT INTO facts (id, predicate, value, ...) VALUES (?, 'enrichment', ...), (3) INSERT INTO edges (id, source_id, target_id, rel_type, ...) VALUES (?, person_id, fact_id, 'HasFact', ...). Do NOT try to insert person_id into facts — the column does not exist.

• Shared enrichment extraction: All web scraping, content extraction, and field validation logic lives in scripts/weave_enrich.py. Both quick_enrich.py and overnight_enrichment.py import from it. When modifying extraction patterns (regex, validation rules, search queries), update weave_enrich.py — never edit duplicated copies in individual scripts.

• Post-migration reference drift: After a backend migration (e.g., LadybugDB → SQLite), ALL reference files must be audited — not just code. schemas.md, gotchas-weave.md, connectors.md, and any file with code examples or schema docs will silently drift. Check every .md in references/ for stale imports, old DB paths, deprecated query languages, and outdated CLI commands. Orphaned reference files (not linked from SKILL.md) should be archived or deleted.

• WeaveDB default path calculation: In scripts/weave_sqlite.py, AGENT_ROOT = Path(__file__).resolve().parents[2] goes up 2 levels from the script, but the skill lives at profiles/indigo/skills/ocas-weave/scripts/. The correct path is parents[3] to reach profiles/indigo/. Using parents[2] points to skills/ which has a stale/empty commons/db/ocas-weave/weave.sqlite. This causes silent write failures — the DB opens but has no data. Verify DEFAULT_DB_PATH resolves to the expected location on first import.

• Function signature drift in pipeline scripts: overnight_enrichment.py called sift_extract_from_pages(name, org, all_results, max_pages=3) but the function signature is sift_extract_from_pages(name, search_results, max_pages=3). The extra org argument shifted all_results into max_pages and max_pages=3 was ignored. Always verify function signatures match when calling shared functions from weave_enrich.py.

• Enrichment field validation too permissive (PATCHED 2026-06-18): validate_field() in weave_enrich.py previously allowed garbage through: sentence fragments as occupations ("As the Editorial Director"), city names as org ("Chicago", "Los Angeles"), single-word generic orgs ("Professional", "Accidents", "per", "newsletter"), partial org names ("was", "Updates", "Product", "San"), invalid locations ("Teague, CP"), junk emails ("[email protected]"), and duplicate fact inserts. Fix applied: org validation now rejects known STATIC_CITIES, generic non-company words, sentence fragments (was/were/been/have/has), values without uppercase letters, and single-character values. Occupation validation requires title-case tokens. Always deduplicate facts before insert.

• SearXNG connection resets under load: During overnight enrichment, SearXNG can return Connection reset by peer or Remote end closed connection without response errors when hit with rapid sequential searches. Add retry logic with exponential backoff (3 attempts, 2s/4s/8s delays) to searxng_search() in weave_enrich.py.

• overnight_enrichment.py duplicate processing: The script's progress tracking does not prevent re-processing contacts that were already enriched in a previous run. If interrupted and restarted, contacts appear in the progress file but may already have facts written. The script also writes duplicate facts (same predicate/value for the same person) when enrich_weave_contact() is called multiple times for the same contact. Always deduplicate after enrichment runs.

• Google outbound sync etag failures: The outbound phase of google_sync.py frequently returns HTTP 400 with "person.etag is different than the current person.etag" for a subset of contacts (observed ~187/587, ~32%). This means Google's contact data was modified externally between the etag fetch and the update push. Workaround: The sync checkpoint prevents re-pushing previously successful contacts, so subsequent runs only retry the failed batch. If failures persist across runs, the checkpoint may need investigation. This is a known rate-limiting/consistency issue, not a data loss risk — inbound sync is unaffected.

Agent-Driven Enrichment Pitfalls (June 2026)

See references/enrichment-agent-driven.md for the full session write-up. Key takeaways:

• Jina Reader blocks LinkedIn: r.jina.ai/linkedin.com/in/... returns SecurityCompromiseError (HTTP 451) with "Anonymous access to domain www.linkedin.com blocked." Workaround: Skip LinkedIn URLs entirely in the Sift phase. Use SearXNG result snippets (title + content) for extraction instead.
• Direct HTTP to LinkedIn returns authwall: LinkedIn redirects to linkedin.com/authwall for unauthenticated requests. Workaround: Same as above — rely on search engine snippets.
• Regex extraction produces sentence fragments: extract_from_content() in weave_enrich.py captures too much text as occupations (e.g., "Prior to Google, Blaise was a Distinguished Engineer", "I am a Senior Product Manager"). Workaround: Apply post-extraction cleaning: reject if starts with known bad prefixes ("I am", "Currently", "Prior to", "Leveraging", "Please send", "Show Details"), reject if >50 chars, reject if person's own name appears in the value, require title-case first letter. See the clean_occupation() pattern in /tmp/batch_enrich_v2.py for a working implementation.
• Wrong-person data on shared pages: When fetching pages that mention multiple people (e.g., event pages, company team pages), regex extraction can capture another person's title/email. Workaround: Reject occupation values containing the contact's own name (means the regex captured a different person's context). Reject emails that don't match the contact's known domain or name pattern.
• Garbage org values from navigation/UI text: Regex captures UI elements like "Pages", "Baseball", "Us", "El", "Save", "User" as org values. Workaround: Maintain a reject set of known garbage org values. Require org to be a proper noun (starts with capital letter, not a common English word). Reject single-character values.
• web_search as primary discovery tool: web_search (Exa AI) returns higher-quality LinkedIn data than SearXNG for professional profiles. The LinkedIn title + description in search results is more reliable than regex extraction from full page HTML. Recommendation: Always run web_search first for each contact, use results for occupation/org/location extraction, then supplement with SearXNG for additional sources. Key pattern: Parse the LinkedIn title + description from web_search results directly — the format is typically "Job Title at Company | LinkedIn" with a description containing location. This avoids the Jina Reader page-fetch step entirely for most contacts.
• Composio LinkedIn requires person_id not username: LINKEDIN_GET_PERSON takes person_id (e.g., yrZCpj2Z12), not vanity username (e.g., jonesabi). There is no name-search tool in the LinkedIn MCP. Workaround: Extract person_id from LinkedIn profile URLs (linkedin.com/in/username → use username to search via web_search, then extract the actual profile ID from the canonical URL or use the username directly with direct HTTP).
• Data quality red flags for org values: Reject org values that are: (1) known city names, (2) single generic words (Professional, Employees, Newsletter), (3) sentence fragments containing verbs like "was"/"were"/"been", (4) values without any uppercase letters, (5) email addresses or URLs, (6) values matching the person's own name.
• google_api.py silent refresh failure (PATCHED 2026-06-20): get_access_token() in google_api.py had two compounding bugs: (1) the credential file stores expiry as a Unix timestamp float (e.g., 1781939144.66) but the code called datetime.fromisoformat(expiry) which throws ValueError on a float; (2) the except Exception: pass silently swallowed the error and returned the expired token without refreshing. The Google People API then returns HTTP 401. Fix: check isinstance(expiry, (int, float)) and use datetime.fromtimestamp() for numeric values, otherwise fall back to fromisoformat(). After fixing, also verify the refresh token itself hasn't been revoked — invalid_grant from the token endpoint means the OAuth consent flow must be re-completed by <operator>.
• google_sync.py unhandled auth failure (PATCHED 2026-06-20): Even after get_access_token() was fixed to raise RuntimeError on invalid_grant, the google_sync.py __main__ entry point had no try/except — it let the exception propagate as a raw traceback and exit code 1. Cron jobs should never crash with tracebacks. Fix: wrap main() in a try/except that catches RuntimeError containing "refresh token revoked" and exits with code 2 and a clean ABORT message to stderr. This distinguishes auth failures (exit 2) from other crashes (exit 1) and avoids noisy cron alerts for a known unrecoverable state.
• Credential file managed by MCP server: The file at <gworkspace-creds>/credentials/<user-google-email>.json is written by the google_workspace MCP server, which may overwrite expiry back to a float after a refresh. Always handle BOTH float and ISO format in get_access_token(). If the MCP server overwrote the file between your json.dump and the next read, the fix is still safe because it handles both formats.
• enrichment_data.py does not exist: There is no enrichment_data.py on disk. Use direct WeaveDB queries and curl for SearXNG health. See the "Agent-Driven Overnight Enrichment" section above.
• web_extract cannot fetch URLs with SearXNG backend: Use curl -s "https://r.jina.ai/URL" instead. This is the reliable page-fetching method in cron/agent context.
• LadybugDB bridge removed: ladybug-bridge-weave.service no longer exists. Skip stop/start bridge steps in the enrichment pipeline.
• WeaveDB.execute() returns dicts, not tuples: db.execute() returns list[dict], not list[tuple]. Use r[0]['column_name'], NOT r[0][0] — the latter raises KeyError: 0.
• Cron invocation may pass a stale runbook: See the ⚠️ CRON INVOCATION section at the top of this skill. The cron job's user message sometimes includes a hardcoded pipeline runbook that references removed components (enrichment_data.py, LadybugDB bridge). Always defer to the skill's own documentation over the runbook in the invocation message. The skill is updated first; the cron message template may lag. If the runbook says "stop the LadybugDB bridge" or "run enrichment_data.py", those instructions are stale — skip them and follow the skill's Agent-Driven Enrichment section instead.
• Heredoc Python in terminal triggers false backgrounding detection: Using python3 << 'EOF' in a foreground terminal() call may be rejected with "Foreground command uses '&' backgrounding". Workaround: Write the script to a temp file (/tmp/weave_batch_enrich.py) via write_file, then run it with python3 /tmp/weave_batch_enrich.py.
• Pre-existing garbage data in persons table: Some contacts have junk occupation/org values from prior bad enrichment runs (e.g., occupation="Save", org="Riegel", org="New", org="St", org="YouTube", org="PI", org="_VOIS", occupation="gram Manager Big Tech Refuge", org="George", org="Donna Karan New York"). Before enriching, scan for and clear known garbage values so COALESCE doesn't preserve them. Common garbage: single-word orgs that aren't companies ("New", "St", "Early", "Los", "Experienced", "Arsenal", "PI", "Converge", "DockerCon", "YouTube", "George"), non-job occupations ("Save", "All Restaurants", "Short Interest", "Building Manager", "gram Manager Big Tech Refuge"), brand-orgs that aren't the person's employer ("YouTube", "Donna Karan New York"), partial org names ("_VOIS").
• Stale org=Google from bad enrichment: Many contacts got org=Google from sync metadata or prior enrichment. Clearing heuristic: keep org=Google only if the person has corroborating data — either an @google.com email address OR both occupation AND location_city populated. Without corroboration, set org to NULL. Same heuristic applies to other major tech companies (Microsoft, Salesforce, Amazon) when there's no email match or other data to confirm.
• Non-person entries in contacts: Some "persons" are actually businesses/services (Doordash, Amazon.com, Resy, Visualping, Wealthfront, Harbor View Plaza). Skip these during enrichment — they have business emails (info@, support@) and no individual professional profile.
• sys.path must use absolute paths in cron scripts: When writing batch scripts to /tmp/, use sys.path.insert(0, '<hermes-home>/profiles/indigo/skills/ocas-weave/scripts') — NOT a relative path like 'scripts'. The cron working directory is the home dir, not the skill dir. Relative paths cause ModuleNotFoundError.
• Empty string vs NULL: The persons table uses both NULL and '' (empty string) for unfilled fields. Your update filter must check BOTH: if not person.get('occupation') catches both None and '' in Python. Don't write separate SQL for IS NULL and = ''.
• Duplicate person records: Some names appear multiple times with different IDs (e.g., two "Abi Jones" records, two "Cameron Moberg" records). Query by name to find all variants and enrich each one. Don't assume ID uniqueness by name. Note: dual-person queries in cron-pipeline-runbook.sql LIMIT 50 may return duplicates that inflate coverage metrics — track by distinct name, not distinct ID, when reporting "both occ+org" counts.
• Subagent enrichment writes may hit wrong DB path: When using delegate_task to spawn enrichment subagents, the subagent receives NO context about the correct DB path by default. If the subagent uses WeaveDB() (which resolves via parents[3]) it lands correctly. But if it uses sqlite3.connect() directly or imports via a relative sys.path.insert(0, 'scripts'), it may hit <hermes-home>/commons/db/ocas-weave/weave.sqlite (stale, 953 persons) instead of .../profiles/indigo/commons/db/ocas-weave/weave.sqlite (canonical, 1052 persons). This produces enrichment facts in the wrong DB that are invisible from the canonical one. Fix: Always include canonical_db_path = '<hermes-home>/profiles/indigo/commons/db/ocas-weave/weave.sqlite' in subagent task context. After subagent completion, verify enrichment facts by ID in the canonical DB.

Support File Map

File	When to read
references/schemas.md	Before any DDL, upsert, or import — Python usage pattern and schema
references/gotchas-weave.md	Before any Weave operation — full gotcha catalog
references/query_patterns.md	Before any weave.query call — SQL templates for all modes
references/connectors.md	Before any Google/Clay sync
references/sqlite-backend-research.md	Storage backend details, migration notes, SQLite schema
references/enrichment-pipeline.md	Overnight enrichment architecture, SearXNG retry pattern
references/enrichment-agent-driven.md	Agent-driven overnight enrichment pipeline architecture, cleaning rules, tool workarounds (June 2026)
references/enrichment-run-2026-06-30.md	Session write-up for June 2026 overnight run — what worked, what didn't, contacts enriched, action items for next run
references/enrichment-write-pattern.md	Exact SQLite write pattern for agent-driven enrichment — three-step (persons UPDATE → facts INSERT → edges INSERT), cron-mode terminal usage, read-back verification
references/cron-pipeline-runbook.md	The correct step-by-step runbook for agent-driven enrichment — modern pipeline (no LadybugDB bridge, no enrichment_data.py), cron-mode terminal usage, confidence scoring guide, read-back verification pattern
references/constraints.md	Full constraint set
references/config-defaults.md	Default config structure
references/self-update.md	Self-update procedure
references/enrichment-data-quality.md	Data quality patterns, garbage categories, validation rules, SearXNG reliability
references/unresolvable-contacts.md	Unresolvable contacts protocol — when to skip (common name, no disambiguator, multiple conflicting profiles), identity resolution ladder, confidence thresholds, log format
references/recovery-weave.md	Recovery contract details
scripts/weave_sqlite.py	SQLite backend module — import WeaveDB from here
scripts/google_api.py	Shared Google OAuth + API helpers — import get_access_token, api_get, api_post, api_patch, PEOPLE_API_BASE from here. All scripts that talk to Google APIs should use this module, not duplicate auth logic.
scripts/migrate_ladybugdb_to_sqlite.py	One-time migration script (already run June 2026)
scripts/migrate_edges_fk.py	FK migration: removes incorrect FOREIGN KEY (target_id) from edges table. Run once; safe to re-run (idempotent).
scripts/weave_enrich.py	Shared enrichment extraction, search, and validation. Contains searxng_search, fetch_page, extract_from_content, validate_field, is_auth_walled, build_scout_queries. Used by both quick_enrich.py and overnight_enrichment.py — do not duplicate this logic in individual scripts.
references/discovery-fallback.md	When Scout sources are degraded/unavailable — SearXNG backoff pattern, DuckDuckGo HTML scrape recipe, page-fetch options, and the no-fabrication defer path. Read before any enrichment run where web_search/SearXNG/LinkedIn MCP are suspect.
scripts/discovery_probe.py	Run at pipeline start to test which discovery sources are live (SearXNG, DDG, notes on web_search/LinkedIn MCP). Decides proceed / fall back / defer.

Visibility

public