Backfilling sources

Backfilling Sources

Backfill a source's changelog history without the remote extraction inference bill (the update workflow's server-side Haiku / tool-loop passes), using the backfill-source dynamic Workflow (and backfill-sweep for several at once). The Workflow wraps the local-ingest primitives — preflight, fetch + extract, /batch upsert, parity rules — in a deterministic harness that owns the window cap, the budget gate, the dedup, and the safety gate, so the disciplines that are fragile when left to prose are enforced in code.

Local Claude Code only. The Workflow fans out agent() sub-agents and relies on a persistent local filesystem (~/.releases/work/) and the CLI's RELEASES_API_* env. It is not deployed to the managed-agent fleet.

When to use

• A source has substantial history to backfill and running the remote update workflow per window is too expensive.
• A remote fetch burned an extraction loop and wrote 0 releases — extracting locally sidesteps the loop.

When NOT to use

• A clean feed / GitHub source — just add it (managing-sources) and let cron fetch it.
• Inside a managed-agent session — this is local-only.
• A publisher opt-out — the preflight gate refuses it (see below).

Cost contract

• Spends: your Claude Code session tokens for the agent() sub-agents, hard-capped by the turn's budget.total (set with a +Nk directive). Extraction runs at Sonnet; the mechanical phases (preflight, run-setup, write, validate, report) run at Haiku.
• Does NOT spend: no server-side extraction (the update workflow's incremental-Haiku / tool-loop passes), no metered Anthropic API bill. POST /v1/workflows/update is never called. /batch runs no AI on insert.
• Always dry-run first (the default) — it maps + estimates and writes nothing.
• When a dry-run is worth it: on index → detail sources, where it enumerates the per-release pages it would pull and shows how many are new vs already-ingested. For a known single-page source it adds little — it can only confirm shape/reachability, not preview record counts (those need extraction), and the workflow already routes single-page through a lighter recon path (no run-setup or known-URL agent on the way to the estimate). Going straight to dryRun: false with a turn budget is reasonable there.

Preflight gate (non-negotiable)

The Workflow runs local-ingest's preflight.ts first and fails closed: refuse (an ai-input=no opt-out — the gate is on ai-input only; ai-train=no alone proceeds) or a persistent unknown stops the run before any fetch or write. conductor.build (Content-Signal: ai-train=no, ai-input=no) is the regression target — it must be refused, via its ai-input=no. Override only with explicit, documented publisher permission.

Launch recipe

Launch by scriptPath, not by name — project .claude/workflows/ scripts are not registered in the Workflow name registry (only plugin-shipped workflows like deep-research resolve by name), so Workflow({ name: "backfill-source" }) errors not found (#1407). The path is repo-relative to the session cwd.

Dry-run one source, review the plan + counts, then commit:

Workflow({
  scriptPath: ".claude/workflows/backfill-source.ts",
  args: { source: "acme/changelog" },
}); // dry-run (default)
Workflow({
  scriptPath: ".claude/workflows/backfill-source.ts",
  args: { source: "acme/changelog", dryRun: false, maxReleases: 50 },
});

Set a turn budget to cap spend (e.g. prefix the request with +300k). Several sources at once:

Workflow({
  scriptPath: ".claude/workflows/backfill-sweep.ts",
  args: { sources: ["acme/changelog", "globex/releases"], dryRun: false },
});

args: source (slug | src_ id | URL), maxReleases (default 50), dryRun (default true), model (sonnet default; haiku for bulk/simple). Sweep takes sources: string[] and resolves each child backfill-source by the same sibling path (override with backfillScriptPath for a non-standard checkout layout).

Re-seed vs. top-up (scrape sources)

A dry-run showing skippedKnown=0 while the source already has rows is a signal, not a success: scrape-ingested releases are stored with url = null, so the workflow's known-URL dedup finds nothing to skip. A dryRun:false run would then write the overlap as new url'd rows alongside the old url-null rows — duplicates, not a gap-fill.

Correct path for the first backfill of an already-ingested scrape source:

1. Hard-delete the existing rows to free the UNIQUE(source_id, url) dedup slot:

   releases admin release delete --source <slug> --hard
   

2. Then run the backfill with dryRun: false.

Truly incremental re-runs (skippedKnown > 0) only happen after the source already carries per-release URLs from a prior backfill.

Caveat: releases tail --json does not expose url or content — only id/title/summary/publishedAt/source/contentChars/contentTokens. To confirm whether existing rows are url-null, use releases admin release get <id> --json.

Extraction altitude (granularity)

The same page can be split at very different altitudes — per feature, per period (month/quarter heading), per version, or one coarse rollup — and left unguided the extractor chooses per-run, so an unchanged page can yield different record counts on different runs. Pin it by setting the source's metadata.granularity once at onboard to one of feature | period | version | rollup. The workflow reads it (via resolve-source) into the extract prompt so the altitude is deterministic across runs, and the Report phase logs an altitude check when the record count runs high relative to the date span (≫4 records per covered month) and the altitude wasn't pinned to feature — surfacing over-splitting instead of letting it pass silently.

After a run

The Workflow records the run under ~/.releases/work/ in an isolated run dir it mints itself (summary per source, cost line grounded in budget.spent(), cross-run sweep report). It deliberately does not use releases admin work start / the shared .current-run pointer — that pointer is global and leaks across concurrent sessions (#1396), so an automated sweep that held it would absorb an unrelated session's mutations. A sweep threads its run dir to each nested source; per-source summaries are namespaced summary-<slug>.md to avoid clobbering. Because no pointer is set, these runs don't appear in releases admin work status. Review the cross-run report (reports/<date>-backfill-sweep.md) for data-quality findings (empty content, thin pages, deferred-for-budget pages). See maintenance-workspace.md → Concurrency.

Before declaring a source "clean" or "recovered" (required)

A dry run's notStored: 0 — or a top-up's skippedKnown matching every extracted URL — is a claim about what extraction found in the window it looked at, not proof the live page has nothing newer. During the 2026-07-23 → 07-28 outage recovery, dry runs across ten sources all reported new=0 and were read as "no content was lost." A commit run on one of those sources then inserted 5 rows, including a changelog entry dated Jul 22 that was visibly live on the source page the whole time (#2168 item 5b).

Before reporting any source "recovered," "up to date," or "nothing missing," fetch the live page yourself — through the same preflight gate above — and compare its newest 3-5 entries against what's actually stored (releases tail <slug> --json or admin release get). This applies whether the conclusion comes from one source or a whole sweep; a clean tool summary is a starting point for the check, not a substitute for it. If the two disagree, the tool number is what's wrong, not the page.

Media and content backfills (server-side, no local extraction)

The local Workflows above rebuild release rows. Four admin routes heal what's already stored — all idempotent, all dryRun-defaulting where noted, none on any cron:

• POST /v1/workflows/backfill-media { sourceId?, all?, limit?, dryRun? } — R2-mirrors media stored before ingest-time mirroring was active (third-party URLs, no r2Key). This is the ONLY path that re-mirrors populated media — the ingest upsert never overwrites non-empty media. Bounded per call; remaining in the report tells you to loop. dryRun default.
• POST /v1/workflows/backfill-video { releaseId?|sourceId?|all?, limit?, dryRun? } — re-runs inline-video detection (Wistia/Loom/Vimeo/YouTube) over stored bodies and APPENDS the type:"video" poster card to media[] (dedup by linkUrl). For rows ingested before #1549.
• POST /v1/workflows/reextract-source { sourceId, snapshotId?, maxWindows?, dryRun? } — re-runs extraction from a stored raw snapshot (released-raw, 90-day lifecycle → 410 snapshot_expired) with no live scrape. Use when parse logic improved and you want history reprocessed without re-fetching.
• POST /v1/workflows/refetch-release { releaseId, url?, dryRun?, force? } — heals ONE release in place from its live page (same rel_ id, title/content/date replaced, AI fields nulled for regen; media replaced only on extractor hit). A stored #fragment URL requires an explicit same-host url; placeholder content is refused, and a >50% body shrink needs force: true. The fix for thin index-scrape rows left by a crawl-mode flip.

Rule of thumb: wrong/missing media → the two media routes; stale extraction with captured raw → reextract-source; one bad row → refetch-release; missing history → the local backfill Workflows above.

Related

• local-ingest — the primitives this wraps (preflight, /batch, parity); use it for one-off interactive ingests.
• parsing-changelogs — the extraction conventions (type, dates, rollups) the extract prompt follows.
• managing-sources — create the org/source first; naming, primary, playbooks.
• maintenance-workspace.md — the ~/.releases/work/ run-recording convention.