Seeding playbooks

Seeding Playbooks

Coordinate bulk creation or enrichment of playbook agent notes across many orgs using parallel sub-agents.

What a playbook is

A playbook is a per-org skill for fetching that org's releases. Same mental model as the global skills in this corpus (parsing-changelogs, managing-sources, etc.) — but scoped to one organization. It tells a future agent (managed or local) how to pull updates from this org: extraction quirks, naming conventions, which sources are canonical, what to skip, where rollups hide.

Global skills teach general patterns; per-source parseInstructions teach source-specific hints; the playbook is the org-level layer between them. When agents fetch any of an org's sources, they should load that org's playbook into context alongside the global skills — the playbook overrides general rules with specific org behavior.

Write notes as instructions to an LLM, not as human documentation. Imperative voice ("Set version=null", "Parse <h2> as version boundaries"), concrete examples from real data, and only observations that change future fetch behavior.

Currently a local-Claude-Code skill. The dispatch pattern below uses Claude Code's Agent tool — that's what's actually wired up. Managed agents (discovery worker, Haiku worker) can spawn sub-agents in principle, but the harness, prompt scaffolding, and rate-limit accounting for managed sub-agent dispatch in this repo haven't been built yet. If/when that gets wired up, the prompts and dispatch shape in this skill port directly.

When to Use

• Batch-populating playbooks for orgs that have sources but no notes
• Re-running the verified workflow on existing playbooks to enrich them with data-grounded observations
• After a wave of new orgs are onboarded and need initial playbook scaffolding

Step 1: Identify Targets

Find orgs that need playbooks. Run this to check coverage:

bun -e "
const orgs = JSON.parse(Bun.spawnSync(['releases', 'admin', 'org', 'list', '--json'], { stderr: 'ignore' }).stdout.toString());
const active = orgs.filter(o => o.sourceCount > 0).sort((a,b) => b.releaseCount - a.releaseCount);
for (const org of active) {
  const playbook = JSON.parse(Bun.spawnSync(['releases', 'admin', 'playbook', org.slug, '--json'], { stderr: 'ignore' }).stdout.toString());
  const status = playbook.notes?.length > 100 ? 'has notes (' + playbook.notes.length + ' chars)' : 'NEEDS PLAYBOOK';
  console.log(org.slug.padEnd(25) + ' sources=' + String(org.sourceCount).padStart(2) + '  ' + status);
}
" 2>/dev/null

This produces a ranked list of orgs with their playbook status. Target orgs showing "NEEDS PLAYBOOK".

Step 2: Gather Source Details

Before dispatching agents, collect source metadata for the target orgs. Each agent needs to know the org's sources, types, URLs, and product structure. Gather this in bulk:

for org in <slugs>; do
  echo "=== $org ==="
  releases admin org get "$org" --json 2>/dev/null | bun -e "
    const d = JSON.parse(await Bun.stdin.text());
    const products = d.products?.map(p => p.name + ' (' + p.slug + ')').join(', ') || 'none';
    console.log('Products:', products);
    d.sources?.forEach(s => {
      const meta = s.metadata || {};
      const parts = [s.slug, 'url=' + s.url, 'type=' + s.type];
      if (meta.feedUrl) parts.push('feed=' + meta.feedUrl);
      if (s.fetchPriority !== 'normal') parts.push('priority=' + s.fetchPriority);
      if (meta.parseInstructions) parts.push('parseInstructions=YES');
      console.log('  ' + parts.join(' | '));
    });
  " 2>/dev/null
done

Step 3: Choose Workflow and Model

Compilation workflow (fast, metadata-only)

• Agent writes notes from source metadata without querying release data
• Good for: bulk scaffolding, low-priority orgs, initial coverage
• Notes are educated guesses — claims about page structure and cadence are inferred, not verified

Verified workflow (thorough, data-grounded)

• Agent queries release data (list <slug> --json) and fetch logs (admin source fetch-log <slug> --json) before writing
• Good for: high-value orgs, scrape sources, orgs with known data quality issues
• Every claim is backed by observed data — version formats, actual cadence, content quality, fetch errors

Model selection

Model	Cost/playbook	Best for
Opus	~$0.07 (compilation) / ~$0.13 (verified)	Top-10 orgs, complex source sets, first-time verified runs
Sonnet	~$0.01 / ~$0.03	Sweet spot for quality/cost. Most thorough output. Use for top-20 verified runs
Haiku	~$0.008 / ~$0.009	Bulk coverage (orgs 20+). Output is usable but may include filler. Cheapest even with higher token count (extra tokens are cached input)

Step 4: Dispatch Sub-Agents

Launch one agent per org, in parallel. Use batches of 10 to avoid overwhelming the system.

Compilation prompt template

Write playbook agent notes for the org "{slug}" and save them using the CLI.

Playbooks are **skills for agents that will fetch from this org**. Write in imperative voice — tell the agent what to do, not what things are.

Notes have three headings: `### Fetch instructions`, `### Traps`, `### Coverage`.

**{Org name}'s sources:**
{list each source with: slug, type, url, and any notable metadata}

Products: {product list or "none"}

**Fetch instructions**: One paragraph per source in imperative voice. Tell the agent what to do ("Set version=null", "Parse <h2> as version boundaries", "No filtering needed"), what to expect (cadence, content quality), and when to skip.

**Traps**: Bullet list with **bolded trigger labels**. Only include things that would cause wasted work or bad data. Include "Don't re-discover" warnings for disabled sources.

**Coverage**: 2-3 sentences. Which sources are canonical, whether there are gaps.

Save by running:
releases admin playbook {slug} --notes-file - 2>/dev/null <<'NOTES'
YOUR NOTES HERE
NOTES

Verify with: releases admin playbook {slug} 2>/dev/null | tail -20

The playbook header regenerates automatically after any source create/update/delete, and the --notes-file PATCH seeds a fresh header on first write — no separate regenerate step needed.

Verified prompt template

Write a **verified** playbook for the org "{slug}".
Unlike a basic playbook, you must do actual research first.

Playbooks are **skills for agents that will fetch from this org**. Write in imperative voice — tell the agent what to do, not what things are.

## Step 1: Gather data (run all of these)

releases admin org get {slug} --json 2>/dev/null
{for each source:}
releases list {source-slug} --json 2>/dev/null
releases admin source fetch-log {source-slug} --json 2>/dev/null

## Step 2: Analyze what you found

Before writing, answer these questions from the data:
- What version format does each source actually use? Cite examples.
- What's the real publish cadence? Count releases per month from dates.
- Are there fetch errors in the logs? What kind?
- Are there releases with missing dates, empty content, or data quality issues?
- **Is the source actually being fetched?** Compare `lastFetchedAt` to the cadence you just measured. An empty fetch-log + stale `lastFetchedAt` (older than ~3× the publish interval) means the cron is silently skipping this source — usually because the change-detector quirk is `unreliable`. Do not write "all releases ingested successfully" or "no errors" in that case; the source is stranded, not healthy.

## Step 3: Write skill-style notes grounded in data

Structure: `### Fetch instructions`, `### Traps`, `### Coverage`.

**Fetch instructions**: One paragraph per source in imperative voice. Tell the agent what to do ("Set version=null", "Parse <h2> as version boundaries"), what to expect (cadence, content quality), and when to skip. Cite version format examples from actual data.

**Traps**: Bullet list with **bolded trigger labels**. Only include things backed by evidence from fetch logs or release data. Include "Don't re-discover" warnings for disabled sources.

**Coverage**: 2-3 sentences. Which sources are canonical, whether there are gaps.

Every claim must cite observed data. If uncertain, say so explicitly.

## Step 4: Save

releases admin playbook {slug} --notes-file - 2>/dev/null <<'NOTES'
YOUR NOTES HERE
NOTES

Verify with: releases admin playbook {slug} 2>/dev/null | tail -20

Dispatch pattern

// Launch up to 10 agents in parallel per batch
Agent({
  description: "Write playbook: {slug}",
  model: "sonnet", // or "haiku" for bulk
  prompt: compiledPromptTemplate,
  run_in_background: true,
});

Step 5: Handle the Parent-Saves Pattern

Sub-agents may be blocked from saving notes via Bash (heredoc permission issues). When this happens:

1. The agent completes analysis and reports its findings in the result
2. The parent agent (you) saves the notes manually:

releases admin playbook {slug} --notes-file - 2>/dev/null <<'NOTES'
{paste notes from agent result}
NOTES

This is a known limitation of subagent permissions. Plan for it — check each agent's result and save manually if needed.

Step 6: Verify Results

After all agents complete, verify coverage in bulk:

bun -e "
const orgs = [{target slugs}];
for (const org of orgs) {
  const proc = Bun.spawnSync(['releases', 'admin', 'playbook', org, '--json'], { stderr: 'ignore' });
  try {
    const d = JSON.parse(proc.stdout.toString());
    const len = d.notes?.length ?? 0;
    console.log(org.padEnd(25) + (len > 100 ? 'OK (' + len + ' chars)' : 'MISSING'));
  } catch { console.log(org.padEnd(25) + 'ERROR'); }
}
" 2>/dev/null

Important: Do not pipe bun | bun in shell for-loops — stdin contention causes silent failures. Use Bun.spawnSync in a single process as shown above.

Record the Run

A batch run makes real prod mutations across many orgs. Leave a durable, cost-aware trail in the per-user ~/.releases/work/ workspace so the work is auditable after the transcript scrolls away — don't let it evaporate into the conversation. The workspace is in the home dir (not CWD) so the trail is the same whether you run from the monorepo or the releases-cli checkout. Full layout and templates: docs/architecture/maintenance-workspace.md.

Local Claude Code only. This assumes a persistent local filesystem. A managed-agent session runs in an ephemeral sandbox whose disk is discarded on teardown — skip run-recording there until the workspace can be synced to durable storage (see the doc's "Local Claude Code only" note). The dispatch pattern in this skill is local-driven today, so this is the normal case.

At the start of a batch, start a run so the CLI's mutation log points at it:

releases admin work start <batch>
# Creates ~/.releases/work/runs/<ts>-<batch>/ and writes a sticky
# ~/.releases/work/.current-run pointer the CLI reads on every later call.
mkdir -p ~/.releases/work/{tasks,reports} # for the batch definition + cross-run report below

(Honors RELEASES_DATA_DIR — substitute $RELEASES_DATA_DIR/work for ~/.releases/work if set.) With a run active, every releases admin … write — including the parent-saves fallback above, which is where most playbook saves land — auto-appends a line to the run's mutations.jsonl. You no longer hand-collect the raw --json outputs. Use work start, not export RELEASES_RUN_DIR: each Bash tool call is a fresh shell, so a one-time export stops logging after the first command, while the .current-run pointer is read fresh on every invocation. (RELEASES_RUN_DIR set inline still wins as a one-off override; releases admin work status shows the active run, work end clears it.) Optionally write the batch definition (targets, workflow, model) to ~/.releases/work/tasks/<batch>.md up front so the run is re-runnable.

After all agents complete, write the judgment layer the CLI can't capture:

1. Per run — write summary.md in the run dir (the path work start printed; work status reprints it): status, per-target result table, cost, and what changed. mutations.jsonl beside it already holds the raw record of every save.
2. Per session — write ~/.releases/work/reports/<date>-<batch>.md with the cross-run pass-rate / cost table and findings worth acting on.

What to capture in the summary and report (these are the data-grounded observations a verified run surfaces):

• Failure modes: Which agents failed to save? Permissions, timeouts, or bad output? Note parent-saves fallbacks.
• Data quality issues found: Broken feeds, empty content, stale lastFetchedAt. File or list these as follow-up fixes.
• Model quality at this tier: Did Haiku produce usable output or need manual cleanup?
• Coverage gaps identified: Missing sources agents flagged — collect as onboarding candidates.