Build cogmap

Build CogMap

This skill is self-contained: the whole pipeline ships inside it, so it runs standalone wherever the skill is installed (no separate checkout required). It re-ingests notes incrementally — only re-extracting chunks whose text changed, re-clustering only new concept names, re-synthesizing insights only when the graph changed — then builds a self-contained HTML artifact.

Locate the pipeline

The engine lives next to this SKILL.md, in scripts/. The entry point is scripts/refresh.py. Resolve it relative to this skill directory — do not hardcode any repo name. If you are unsure of the absolute path, glob for **/build-cogmap/scripts/refresh.py.

Everything else resolves automatically via scripts/cogmap_paths.py (using __file__), so you never pass absolute paths on the command line.

build-cogmap/            <- this skill
  SKILL.md
  scripts/               engine: refresh.py + stage scripts
  assets/demo/           bundled demo corpus + prebuilt artifacts (demo-only)
  requirements.txt

The workspace (where the user's data lives)

Runtime data is written to a project-local workspace, never into the skill folder. A real run defaults to an empty ./cogmap under the current working directory:

cogmap/                  <- the workspace (in the user's working dir)
  sources/    the user's notes — any .md / .txt files (optional .onex helper)
  work/       extractions + resolved + insights + cache (LLM artifacts)
  output/     knowledge-base-viz.html + knowledge-base-viz-data.json  <- deliverable

The synthetic corpus is never copied into a real workspace. To render it explicitly, run refresh.py --demo --no-open; this creates and uses a separate ./cogmap-demo workspace. Never add user notes to cogmap-demo/, and never copy sample-notes.txt into cogmap/sources/.

Run refresh.py from a stable directory (e.g. the user's project root) so the ./cogmap workspace is predictable. On startup refresh.py prints the resolved workspace paths — use those when reporting the deliverable location. To relocate the workspace explicitly, set COGMAP_APP (or the finer COGMAP_SOURCES / COGMAP_WORK / COGMAP_OUTPUT; legacy OSLER_* names still accepted).

Environment rules

• Python 3.10+ (no third-party deps for .md/.txt). olefile (pip install -r requirements.txt) is only needed for the optional --from-onex OneNote converter. On Windows use python3.12; elsewhere python/python3.
• Always set PYTHONIOENCODING=utf-8 before running (Unicode-heavy notes).
• Sub-agents must read batch JSON with json.load, not a file-view tool (batches exceed 20KB and get truncated).

Source of truth

The pipeline ingests every .md / .txt file in the workspace sources/ (any filename; README.md is ignored). It works with notes from any tool — Markdown/plain-text exports from Obsidian, Notion, Logseq, Apple/Google Notes, journals, meeting notes, etc. The chunker auto-detects prose-paragraph vs one-entry-per-line format, and picks up a date near the top of each note (2025-03-14, March 2025, …) for the timeline; otherwise reading order is used.

OneNote is supported via an optional converter: refresh.py --from-onex best-effort extracts any .onex in sources/ to clean text first (OLE2 package, only ~66% of prose recoverable — warn the user before relying on it; a clean .md/.txt export is always higher fidelity).

Adding the user's notes

When the user wants to add notes, put them in the workspace sources/ and refresh:

• If they give a path, copy the file(s) into <workspace>/sources/.
• If they paste text, write it to a new <workspace>/sources/<slug>.md (or .txt).
• Encourage a date near the top of each note for a chronological timeline.
• Use only the user's content in this real workspace. If the exact bundled sample-notes.txt is present, the refresh stops instead of generating a mixed graph.

Then run the loop below. See sources/README.md in the workspace for the user-facing guide.

The operational loop

refresh.py is a resumable state machine:

• exit 0 — done. Report the final counts and tell the user to hard-refresh.
• exit 10 — an LLM stage is required. Details are in the workspace at work/refresh_cache/action.json ({"action": "extract"|"resolve"|"synth", ...}; all file paths inside are absolute). Handle the stage (see below), then re-run the same command. For extraction, finish every listed pending batch and re-run once; the pipeline validates all outputs before advancing. Repeat until exit 0.

Run it (from the user's working dir; paths self-resolve):

PYTHONIOENCODING=utf-8 python <skill>/scripts/refresh.py --with-resolve --with-synth

Flags:

• --demo — render the bundled synthetic corpus in a separate ./cogmap-demo workspace. This is for product evaluation only, never real note ingestion.
• --with-resolve — on concept-set change, cluster new names into existing/new canonical concepts (best quality). Without it, assemble uses a lossless singleton fallback (new concepts appear un-merged).
• --with-synth — on graph change, refresh insight narratives + synthesis cards. Without it, existing insights are reused. If resolution is stale, this requires --with-resolve or an explicit --skip-resolve; synthesis never runs ahead of an unresolved resolution gate.
• --from-onex — best-effort re-extract .onex → clean text first (~66%).
• --skip-resolve, --skip-synth — force-skip those gates.
• --no-open — don't auto-open the finished HTML (also via COGMAP_NO_OPEN=1).
• --publish-github-pages — after a successful build, publish the visualization to a gh-pages branch and configure GitHub Pages when gh is authenticated. Optional controls: --publish-remote <name> (default origin), --publish-branch <name> (default gh-pages), --publish-path <path> for a subdirectory, --publish-no-push for a local dry run, and --publish-no-enable-pages to skip Pages API configuration.

Default recommendation: --with-resolve --with-synth so a real content change yields fully-merged concepts and fresh insights. Read action.json after every exit-10, handle the stage, re-run. Never hand-edit pipeline outputs.

Handling the LLM stages

Each stage below can be run by dispatched sub-agents or by you directly. Extraction is the only parallel stage. Follow action.json.orchestration: dispatch only the listed pending batches, run no more than max_concurrency at once, wait for every listed batch to finish, and then re-run refresh.py once. If more batches remain, use bounded waves rather than launching unbounded agents. Resolve and synth are single, sequential higher-reasoning passes and start only after extraction has fully validated.

---

Stage 1 — Extraction (action == "extract")

The pipeline packs small and medium deltas into as few agent calls as safely possible. Large deltas adapt toward roughly 8–10 batches, bounded by safe character and chunk limits. action.json exposes the selected policy under batching and includes:

• batches: only missing or invalid batches to run. Each has input, output, chunk_ids, batch_id, status, and actionable diagnostics.
• completed_batch_ids: valid outputs already retained; do not run them again.
• progress: total, valid, and pending counts.
• orchestration.max_concurrency: the safe parallel dispatch bound (at most 8).

For each listed batch, read its input file and write its output file. Preserve the manifest and valid output files; never recreate the delta directory yourself. Prompt / task (fill INPUT/OUTPUT):

Read the JSON array at INPUT using json.load (do NOT use a view tool — it truncates). Each element is a note chunk: {chunk_id, order, source, heading, date, text}. Extract a knowledge graph grounded only in this text. Write a single JSON object to OUTPUT with exactly these keys:

• concepts: [{name, type, definition, aliases:[...], chunk_ids:[...]}] — type ∈ Technology|Method|Concept|Organization|Person|Product|Metric|Risk|Theme. Every chunk_ids value MUST be a chunk_id present in this batch.
• claims: [{text, status, concepts:[names], chunk_id, date}] — status ∈ Observation|Prediction|Decision|Question|Tension.
• relations: [{source, target, type, evidence_chunk_ids:[...], rationale}] — type ∈ enables|causes|depends_on|part_of|refines|competes_with|contradicts|relates_to. source/target are concept names that appear in concepts. Deduplicate names within the batch; use canonical noun-phrase names. Only reference chunk_ids from this batch. Output valid UTF-8 JSON, nothing else.

When all listed outputs are written, wait for all extraction agents and re-run refresh.py once. The pipeline decodes each output as UTF-8, parses JSON, and validates the exact top-level schema, required fields/types, enums, concept references, and batch-local chunk/evidence IDs. Valid completed outputs are skipped on subsequent runs. Invalid outputs remain pending in action.json with diagnostics and must be overwritten by re-running only those batches; they are never ingested or silently discarded.

Stage 2 — Incremental resolve (action == "resolve")

action.json has existing (work/v3_resolved.json), new_names (work/refresh_cache/resolve_new_names.json), uncovered (count). Handle with a single higher-reasoning pass:

Read v3_resolved.json (existing canonical concepts: canonical, category, members:[raw names]) and resolve_new_names.json (new raw names not yet covered). For EACH new name, either add it to the members of the best-matching existing concept, or create a new concept with canonical (clean noun phrase), category, and members. For category: reuse an existing category string exactly when one fits; otherwise coin a concise thematic category (a few words). Aim for ~5–10 balanced categories overall — enough to form meaningful Minard lanes, not one lane per concept. Do not drop or duplicate any name; every new name ends up in exactly one concept. Write the merged result back to v3_resolved.json as a JSON object with a top-level concepts key — exactly {"concepts": [{"canonical": ..., "category": ..., "members": [...]}, ...]} (NOT a bare array). Prefer merging near-duplicates over creating singletons.

Then re-run refresh.py --with-resolve.

Resolution is sequential and is never dispatched until every extraction batch is valid and ingested.

Stage 3 — Synthesis (action == "synth")

action.json has synth_input (work/v3_synth_input.json). Handle with a single higher-reasoning pass:

Read v3_synth_input.json (keys: concepts, evolves_into, enables, contradictions, existing_insight_candidates, categories). Reason over the resolved graph and write v3_insights.json: polished insight objects plus cross-cutting synthesis cards that connect ≥2 concepts across categories and surface an emergent, non-obvious pattern (idea evolution, tension, or convergence). Each item must reference real concept ids/canonicals from the input so provenance derives deterministically. Keep narratives tight and grounded in the supplied graph.

Then re-run refresh.py --with-synth.

Synthesis is sequential after resolution; do not overlap it with extraction or resolution.

---

When done

On exit 0, report the final line (DONE. chunks=… tracked-extracted=…). The finished output/knowledge-base-viz.html is auto-opened in the default browser (disable with --no-open or COGMAP_NO_OPEN=1). If it was already open, tell the user to hard-refresh the browser (Ctrl+Shift+R) — for large corpora the embedded page can be 1 MB+ and can't be reloaded from the agent side.

If the user asks to share or publish the map publicly, re-run the final refresh with --publish-github-pages from a GitHub-backed repository. The publisher copies knowledge-base-viz.html to index.html on the Pages branch, includes the data JSON and a small manifest, pushes the branch, and prints the Pages URL when it can parse the GitHub remote. If automatic Pages enablement fails, report the printed warning and the branch it pushed so the user can enable Pages manually.

Notes

• A first real run creates an empty cogmap/ workspace and exits with instructions until user notes are present.
• refresh.py --demo creates a distinct cogmap-demo/ workspace seeded from assets/demo/. The demo and real corpus never share sources, state, or output.
• A real refresh rejects the exact bundled sample-notes.txt so sample concepts cannot silently leak into a user's graph.
• Correctness mechanism: v3_aggregate.py filters extractions by the current chunk IDs, so stale extractions for edited/removed chunks are auto-dropped. Chunk IDs are content-only (position-independent), so mid-file edits don't cascade.
• Extraction archives use deterministic batch IDs, so retrying ingestion replaces the same archive instead of duplicating graph evidence or edge weights.
• Engine scripts: scripts/refresh.py (orchestrator), v3_prep.py, v3_aggregate.py, v3_assemble.py, build_v2.py, extract_onex.py.