Host test memory leak hunting
Skill cardstack/boxel/.claude/skills/host-test-memory-leak-hunting
Hunt down memory leaks in the @cardstack/host Ember test suite using V8 heap snapshots over CDP. Use when host CI shards OOM, when MEMPROBE output shows used-heap climbing across tests, or when adding a new feature that holds DOM/closure state across the test lifecycle. Documents the local probe loop, the snapshot-analysis scripts, and the catalog of leaks already found.From its SKILL.md
npx -y skills add cardstack/boxel --skill host-test-memory-leak-huntingAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
9.6 KB, as published. Nobody here has run it
Host test memory leak hunting
The host test suite is large enough that a per-test leak as small as 5 MB will OOM a CI shard before the suite finishes. This skill is the playbook for finding and validating fixes. It exists because a single leak hunt can take days if you don't have the loop tightened — Chrome heap snapshots are slow, big, and ambiguous unless you reduce them to retainer paths.
See known-leaks.md for the catalog of leaks found so far and the patterns that cause them. Read it first — most "new" leaks rhyme with one of those.
When to use
- CI host-test shard OOMs (V8
Reached heap limit Allocation failedin the test log). - Local
pnpm testshowsMEMPROBE used=climbing every 10 tests. - New feature added a service, modifier, or globalThis closure that captures the test's
owner. - After landing a fix that is supposed to flatten the slope — verify before declaring victory.
What's already in the suite
Two pieces of permanent instrumentation in packages/host/tests/helpers/setup-qunit.js:
- Forced GC every test (requires Chrome
--js-flags="--expose-gc"). Without this V8's opportunistic GC can't keep up and the heap drifts toward the 4GB ceiling regardless of leak size. MEMPROBElog line every 10 tests withused=/total=/app_instances=/alive=/destroying=/destroyed=— visible in CI output and used by the snapshot runner to trigger snaps.
Three diagnostic scripts in packages/host/scripts/ (only used during a hunt):
heap-snapshot-runner.js— CDP client. Watches Chrome console forPROBE t=N; at configured indices, takes a heap snapshot and streams chunks to disk (joining blows V8's max string length past ~300MB).snapshot-diff.js— top-N constructor counts/retained-sizes that grew between two snapshots.snapshot-retainers.js— backward BFS from target nodes to GC roots. Flags:--type=<native|object|closure|string>,--min-size=N,--strong(skipsweak/shortcutAND WeakMap "part of key" internal edges; use this — it's the difference between a misleading WeakMap path and the real strong retainer).snapshot-by-class.js— like snapshot-diff but truncates long names so output is scannable.snapshot-diff-stream.js/snapshot-retainers-stream.js— use these when a snapshot file is >500 MB. Same flags and output as the non-streaming versions; they usestream-jsonto avoidfs.readFileSync('utf8')(which throwsERR_STRING_TOO_LONGabove V8's ~512 MB cap). The retainers-stream variant also buffers nodes/edges into a growableFloat64Arrayso huge snapshots don't hitInvalid array lengtharound 100M+ items. Just swap the script name; all arguments are identical.
See packages/host/scripts/HEAP_PROBE.md for invocation reference.
The probe loop
The dev stack must already be up via mise run dev-all, with Chrome launched with --remote-debugging-port=9333 --js-flags="--expose-gc" --enable-precise-memory-info. (You can launch a dedicated Chrome on macOS with /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9333 --js-flags="--expose-gc" --enable-precise-memory-info --user-data-dir=/tmp/chrome-leak-probe.)
1. Pick a fast filter
The test filter you choose dominates iteration time. Use a module that exercises the suspect code path AND completes ~1 second per test. Good defaults:
card-basics— 99 tests, ~1s/test, exercises card-api + Box trees (hits the most common leak shape)Integration | Store— good for store/card-api-related leaks; reproduces ~20 MB/test for store service leaks thatcard-basicsmisses (~1.1 MB/test)Integration— slower, broader, use only when narrower filters miss the signal- A specific module name from a failing CI shard
Avoid filters that include realm-indexing-heavy tests (e.g. those that intentionally trigger errors) — those run at ~60s/test and crush iteration.
Important: if card-basics doesn't reproduce the expected slope, try a heavier filter. Some leaks only trigger when specific service code paths are exercised (e.g. StoreService.setup() only fires when the store is actually used).
2. Start the snapshot runner
rm -f /tmp/snap-t*.heapsnapshot /tmp/snap-runner.log
SNAPSHOT_AT="10,50,90" \
nohup node packages/host/scripts/heap-snapshot-runner.js > /tmp/snap-runner.log 2>&1 &
Snapshots at t=10 (warm) and t=50 give a clean delta over 40 tests. Add t=90 to confirm the t=10→t=50 slope continues (rules out one-time allocations).
SNAPSHOT_AT values must be multiples of 10 — the runner triggers on MEMPROBE log lines, which only fire every 10 tests. Values like 3 or 15 will be silently missed.
3. Open a fresh test tab
ENCODED=$(node -e 'console.log(encodeURIComponent("https://localhost:4200/tests/index.html?hidepassed&filter=card-basics"))')
curl -sX PUT "http://localhost:9333/json/new?${ENCODED}"
Important: close any prior /tests/ tabs first — the runner picks the first matching tab and will get stuck on a stale one. The runner exits after writing the last snapshot in SNAPSHOT_AT.
4. Read the MEMPROBE lines
The runner echoes every MEMPROBE line as it streams from the browser. Look at the trajectory:
used=climbing linearly = leak. Per-test slope =(used_t90 - used_t10) / 80.app_instances=> 0 between tests =App._applicationInstancesleak (see known-leaks.md).destroyed=N alive=0with N > 0 =willDestroyran but_unwatchInstancewas skipped (super.willDestroy threw).
5. Diff the snapshots to identify what grew
node --max-old-space-size=16384 packages/host/scripts/snapshot-diff.js \
/tmp/snap-t10.heapsnapshot /tmp/snap-t50.heapsnapshot | head -40
The top of the output sorted by retained-size delta tells you what's accumulating. Watch for:
+N copiesofstring::define("https://cardstack.com/base/...")— a Registry/factory chain is pinning fresh-per-test card-api module sources. Almost always points back toApp._applicationInstancesretention.+N copiesofJSArrayBufferData(native) — usually downstream of the above (each pinned ApplicationInstance brings ArrayBuffers).+NBox / FieldComponent / Glimmer-internal counts — Box tree retention.+Nofnative::DOMTimer— orphanedsetInterval/setTimeoutfrom async service code that ran after the service was destroyed (see known-leaks.md #5).
6. Trace the retainer chain
Pick a high-count constructor from the diff and trace it:
node --max-old-space-size=16384 packages/host/scripts/snapshot-retainers.js \
/tmp/snap-t50.heapsnapshot '^Box$' \
--max=10 --depth=30 --type=object --strong
Always pass --strong. Without it, the shortest path runs through weak edges and WeakMap "part of key" internal edges — neither of which actually retain the target. With --strong, you see the real GC-root path. The retainer signature (the chain of edge labels) tells you what to fix.
If the chain ends at Window.@cardstack/host → some service or registry, fix the leak there. If it ends at synthetic root → (GC roots) → some array, you're likely looking at a global Set/Map that needs a WeakMap or an explicit cleanup.
7. Apply the fix and re-run from step 2
If the slope flattens and the offending constructor count stops growing, you're done. Save a memory entry with the retainer chain and the fix.
Pitfalls
getContext()returns undefined atQUnit.testDonebecauseunsetContext()runs inteardownContextbefore the owner is destroyed. Don't snapshotctx.ownerin testDone — capture inhooks.afterEachof the test module, or readgetApplication()._applicationInstancesdirectly.- WeakMap edges. Heap snapshots show WeakMap key→value as a normal-looking edge.
snapshot-retainers.jsskips them when--strongis passed; otherwise you'll chase ghosts. - V8 max string length on big snapshots.
JSON.parse(fs.readFileSync(snap, 'utf8'))works up to ~500MB, butchunks.join('')blows up around 300MB. The runner streams chunks straight to disk for this reason. The analysis scripts (snapshot-diff.js,snapshot-retainers.js) also hit this limit — if snapshots exceed ~500MB, take them earlier in the run (lowerSNAPSHOT_ATvalues) to keep file sizes manageable. - GC timing. The
globalThis.gc(); globalThis.gc()double-call in setup-qunit is intentional — V8 sometimes needs a second pass to actually collect. If a snapshot still shows the suspect, re-snap after a short wait. - Stale tabs. Each
/json/new?<URL>call creates a NEW tab. The runner picks the first one. Close stale tabs before starting the runner.
Validating the fix
A clean fix is one that:
- Makes the per-test slope flat in steady state (compare t=50→t=90, not t=10→t=50 — the early region warms up caches).
- Drops the leaked constructor count delta to zero (or single digits — V8 caching introduces a small floor).
- Still passes the full host-test shard 3 (the historical canary for memory pressure) under the original
shardTotal: 20config.
If you bumped CI shard count as a temporary mitigation, revert that bump in the same PR as the fix.