Cloud cascade debug

AILANG Cloud Cascade Debugger

After ailang publish of a package whose dependents should auto-bump (the M-PKG-AUTONOMOUS-CASCADE-SAFE pipeline), trace the message through: registry validator → ailang-cascade Pub/Sub topic → coordinator subscriber → task dispatch → Cloud Run Job execution. Surfaces the first failing hop with timestamps + log lines.

When to use

User says any of:

• "auto-cascade didn't work" / "auto-cascade didn't fire"
• "why didn't multivac pick this up"
• "the dependents didn't get bumped"
• "check the cloud logs for the cascade"
• "I had to manually republish — what happened cloud-side?"

Quick triage (60-second sanity check)

# Which project did the publish actually target?
echo "AILANG_CLOUD_PROJECT=$AILANG_CLOUD_PROJECT"
echo "AILANG_TOPIC_PREFIX=$AILANG_TOPIC_PREFIX"

If AILANG_CLOUD_PROJECT=ailang-multivac-dev (or *-dev), cascade messages went to the dev project's topic — production coordinator never sees them. That alone explains "auto-cascade didn't fire" for prod consumers. Keep investigating in the dev project then.

The pipeline (what should happen)

1. ailang publish [email protected]
   ├─→ POST /publish to ailang-registry-validator
   │   (project: ailang-registry; logs in europe-west1)
   │
   └─→ Pub/Sub publish to ailang-cascade topic
       (project: $AILANG_CLOUD_PROJECT; topic: $PREFIX-cascade)
           │
           └─→ Push subscription delivers to:
               https://<coordinator>/pubsub/push
                   │
                   └─→ Coordinator creates task per dependent
                       Looks up agent for inbox "pkg:sunholo/<dep>"
                       Dispatches to Cloud Run Job with env:
                         AILANG_TASK_ID=task-XXX
                         AILANG_AGENT_ID=<resolved agent>
                         (etc., see internal/dispatch/cloudrun/dispatcher.go:117+)
                           │
                           └─→ Job starts, runs the bump, publishes
                               completion to ailang-completions

Any hop can fail silently — the publish CLI prints "Cascade-topic notification published" before any subscriber acks, so success messages don't mean a dependent actually bumped.

Workflow

Step 1 — Identify the publish you're investigating

Get the timestamp + project from the publish output (or from ls -lt ~/.ailang/cache/registry/<vendor>/<name>/):

✓ Published sunholo/[email protected]
→ Cascade-topic notification published for sunholo/motoko_ext_omnigraph
  (root=sunholo/[email protected], class=A)

The cascade message IDs printed there (e.g. msg_20260511_211114_88b4f765) are the ones to grep for. Convert local time to UTC for log queries.

Step 2 — Check the validator did receive the publish

gcloud logging read \
  'resource.type="cloud_run_revision" AND
   resource.labels.service_name="ailang-registry-validator" AND
   httpRequest.requestMethod="POST" AND
   timestamp>="2026-05-11T19:00:00Z" AND
   timestamp<="2026-05-11T19:30:00Z"' \
  --project ailang-registry --limit 20 \
  --format="value(timestamp,httpRequest.requestUrl,httpRequest.status)"

Expected: HTTP 200 to /publish at the time of your publish. If 4xx/5xx, the publish failed before cascade messages were emitted — investigate validator logs directly.

Step 3 — Confirm cascade messages were emitted

The publish CLI logs each cascade message it emits. Cross-check against what landed in Pub/Sub. The topic is <PREFIX>-cascade:

PROJECT="${AILANG_CLOUD_PROJECT:-ailang-multivac-dev}"
PREFIX="${AILANG_TOPIC_PREFIX:-ailang-dev}"

gcloud pubsub topics list --project "$PROJECT" \
  --filter="name~cascade" --format="value(name)"

If the topic doesn't exist for that project: the cascade publisher silently no-ops (see cmd/ailang/pkg_publish.go::newCascadePublisher). The publish CLI may still print "Cascade-topic notification published" because it prints from the legacy-inbox-write path even when the cloud publisher is nil — confusing but documented.

Step 4 — Find the coordinator that subscribes

gcloud pubsub subscriptions list --project "$PROJECT" \
  --filter="topic~cascade" \
  --format="value(name,pushConfig.pushEndpoint)"

Push endpoint will be a Cloud Run URL like https://ailang-dev-coordinator-XXX-ew.a.run.app/pubsub/push.

Step 5 — Check the coordinator received and processed each cascade message

# Substitute the time window of your publish
gcloud logging read \
  'resource.type="cloud_run_revision" AND
   resource.labels.service_name="ailang-dev-coordinator" AND
   timestamp>="2026-05-11T19:10:00Z" AND
   timestamp<="2026-05-11T19:30:00Z" AND
   (textPayload=~"cascade" OR textPayload=~"task-" OR textPayload=~"failed")' \
  --project "$PROJECT" --limit 50 \
  --format="value(timestamp,textPayload)"

Look for log lines:

Pattern	Meaning
Created task task-XXX (... inbox: pkg:sunholo/...)	coordinator received the cascade message
Skipping message <id>: no agent registered for inbox "..."	no route; NOT dispatched, left unread for triage (current behaviour)
Cloud dispatch: task task-XXX → Cloud Run Job (agent: , ...)	agent: empty — pre-2026-08-25 behaviour; dispatched anyway and failed on arrival
task task-XXX → failed (error=...)	the job ran and exited with an error
task task-XXX already in terminal state "failed", skipping	repeated retries from Pub/Sub redelivery
(no logs at all)	coordinator scaled to zero AND push subscription couldn't wake it (rare)

Step 6 — Unrouted inbox: check this FIRST

The coordinator's agent registry is loaded from ~/.ailang/config.yaml (or the GCS config on the cloud instance). Since M-COORDINATOR-INBOX-WILDCARDS an inbox with no match is refused, not dispatched — the message is left unread for triage.

Inbox patterns work (a trailing * only — this was documented here before it was implemented, so older notes may mislead):

coordinator:
  agents:
    - id: pkg-motoko-ext-family
      label: "Package family: sunholo/motoko_ext_*"
      inbox: "pkg:sunholo/motoko_ext_*"   # trailing * = family pattern
      capabilities: [code, test, docs]

Precedence: exact inbox: wins, then longest matching prefix (pkg:sunholo/motoko_ext_* beats pkg:sunholo/* beats pkg:*), then no dispatch.

Check the registry spelling, not the directory spelling. Packages are published with underscores and their directories use hyphens, and FormatPackageInbox does no normalization — sunholo/motoko_ext_abi routes to pkg:sunholo/motoko_ext_abi, never pkg:sunholo/motoko-ext-abi. Ten ailang-parse tickets were lost to exactly this.

grep -o 'inbox: "pkg:[^"]*"' config/config.cloud.yaml | sort -u

Pre-2026-08-25 the task was dispatched with an empty AgentID and the executor's startup check (cmd/ailang/coordinator_cloud.go:171) failed them all, posting the failure to inbox "" where nothing could read it. See docs/internal/cloud-coordinator-config.md.

Step 7 — If the job ran and failed for another reason

Pull the failed job's logs by task ID:

TASK_ID="task-cb56ee02"
gcloud logging read \
  "resource.type=\"cloud_run_job\" AND
   labels.\"run.googleapis.com/execution_name\"=~\"$TASK_ID\"" \
  --project "$PROJECT" --limit 100 \
  --format="value(timestamp,textPayload)"

Or by execution name (faster):

gcloud run jobs executions list --project "$PROJECT" \
  --region europe-west1 --limit 20 \
  --format="value(metadata.name,status.completionTime,status.conditions[0].message)"

Common failure modes (cheat-sheet)

Symptom	Likely cause	Fix
Publish prints "Cascade-topic notification published" but no coordinator logs	AILANG_CLOUD_PROJECT points to a project with no *-cascade topic	Check env vars; the cloud publisher silently no-ops when topic absent
Coordinator logs Created task ... (agent: , inbox: pkg:...) then failed (error=AILANG_AGENT_ID required)	No agent registered for pkg:* inboxes in coordinator config	Add package-cascade-bumper agent entry
Coordinator hasn't logged ANYTHING since hours before the publish	Cloud Run revision crashed; push delivery failing silently	gcloud run revisions list; check latest revision health
Tasks created but never dispatched (Cloud dispatch: published task ... to Pub/Sub only (no dispatcher, ...))	cloudDispatcher not registered (CLI didn't init cloudrun.NewDispatcher)	Restart coordinator with proper env (AILANG_CLOUD_REGION, etc.)
Job dispatched but immediately fails with AILANG_AGENT_ID required	Same as the empty-agent case above	Same fix
Job dispatched, runs for hours, never publishes completion	Job hung mid-execution; check Cloud Run Job execution logs	gcloud run jobs executions describe

One-shot triage script

.Codex/skills/cloud-cascade-debug/scripts/triage_cascade.sh \
  <vendor/name>@<version> [time-window-hours]

Bundles steps 2-5 into a single output. Defaults to a 1-hour window ending at NOW. Pass --project to override the project (default: $AILANG_CLOUD_PROJECT).

Notes

• Production project = ailang-multivac (no -dev suffix); dev project = ailang-multivac-dev. Check AILANG_CLOUD_PROJECT carefully — local laptops typically point at dev.
• The validator service lives in ailang-registry (separate project) but the cascade pipeline lives in whichever project the publisher targets via AILANG_CLOUD_PROJECT.
• All of this is europe-west1.
• The cloud publisher is best-effort — a failure to write to the cascade topic does NOT abort ailang publish (the package still publishes). Cascade is "fire and forget" by design.