Webhook delivery

Webhook Delivery

Webhooks are the highest-leverage trust surface a developer tool has: they run inside the customer's infrastructure, and every flake is a support ticket on their side. The bar is Stripe. This skill gets a design to that bar without a year of incidents teaching it the hard way.

Before you start

Check for .agents/devtool-context.md and read it if present (it tells you the API shape, stack, and scale, all of which change the design). If absent, ask:

1. What events would customers subscribe to, and what triggers them?
2. Expected volume: tens a day or thousands a second?
3. Build vs. buy appetite: is a hosted sender (Svix, Hookdeck) on the table, or is this in-house?

On build vs. buy: below serious scale, Svix is the correct default answer; Clerk, Resend, and Brex ship webhooks this way. Recommend it unless the user has a real reason (data locality, cost at volume, it is their product). If they buy, skip to the envelope and consumer-docs sections. Those decisions remain theirs either way.

The design, in order of consequence

Work through these with the user and produce a written design doc plus schema/code artifacts as you go. Positions below are defaults; deviate only with a reason.

1. Event envelope

Every event shares one envelope. Design it once, version it forever:

{
  "id": "evt_7f3a2b9c1d",
  "type": "invoice.paid",
  "api_version": "2026-08-01",
  "occurred_at": "2026-08-10T12:00:00Z",
  "data": { "object": { /* the resource */ } }
}

• Event IDs are the consumer's idempotency key. Prefix them (evt_); see api-keys for why prefixes matter everywhere.
• type uses object.verb naming (invoice.paid, not PaidInvoice). Keep the taxonomy in one registry file. The event-taxonomy skill governs this; the webhook type field and the analytics event names should not drift apart.
• Thin vs. fat payloads: default fat (full resource in data). It saves the consumer an API round-trip and Stripe-conditioned developers expect it. Go thin (IDs only, consumer fetches) when payloads carry data too sensitive to spray at customer servers, or when out-of-order delivery makes stale snapshots dangerous. Never mix per-event-type; pick one.

2. Signatures

Sign every delivery. HMAC-SHA256 over {timestamp}.{raw_body}, secret per endpoint, sent as a header:

Webhook-Signature: t=1723291200,v1=5257a869e7...

• Timestamp inside the signed content kills replay attacks; use a tolerance window of 5 minutes on the consumer side.
• The v1= scheme tag is what lets you rotate algorithms later without breaking anyone. Allow multiple signatures in one header during secret rotation.
• Endpoint secrets are whsec_-prefixed, shown once, rotatable from the dashboard with a 24h overlap where both secrets sign.
• Document consumer-side verification with copy-pasteable code in every SDK language you ship. Constant-time comparison, always.
• The Standard Webhooks spec codifies this same approach with different mechanics: three headers (webhook-id, webhook-timestamp, webhook-signature) signing {id}.{timestamp}.{payload}, signatures as v1,-tagged base64. Adopting it wholesale buys you off-the-shelf verification libraries; if you ship the Stripe-style single header above instead, document it as your own scheme and don't mix the two.

3. Delivery semantics: at-least-once, unordered

Say this out loud in the docs, because it's the contract consumers must build against:

• At-least-once. Exactly-once delivery over HTTP is a lie; duplicates happen (a timeout after the consumer processed but before you recorded it). Consumers dedupe on event id.
• Unordered. Retries reorder events by construction: invoice.paid can arrive before invoice.created. Do not promise ordering; instead put enough state in each event (fat payload with the resource's current state, plus occurred_at) that the consumer can handle either order. If a consumer truly needs order, they should treat the webhook as a "poke" and fetch current state. Say so in the docs.

4. Retries

• Queue-backed sender (whatever queue the stack already has; SQS, Cloud Tasks, or pg-boss on Postgres is fine at small scale). Never send from the request path of the triggering action.
• Exponential backoff with full jitter. First retry fast (~30s, transient blips are common), then spread out: roughly 30s → 5m → 30m → 2h → 5h → 10h, which gives about a day of retries over 6-8 attempts. Read references/retry-schedule.md when finalizing the exact schedule and failure-classification rules (which HTTP responses retry vs. hard-fail).
• 10s delivery timeout. Consumers should ACK with 2xx before doing work. Put that in the docs.
• Endpoint auto-disable: after sustained failure (all retries exhausted for N consecutive events or ~5 days), disable the endpoint and email the customer. Hammering a dead endpoint forever wastes your queue and their logs.

5. Dead letters and replay

• Exhausted deliveries land in a dead-letter state, retained ≥30 days.
• Replay is the feature that turns webhook incidents from disasters into non-events. Dashboard button + API endpoint to redeliver any event or all failures in a time range. Resend and Stripe both expose per-event "resend"; consumers use it constantly during their own outages.
• Replayed events keep their original event id (consumer-side dedupe must keep working) but a fresh delivery timestamp/signature.

6. Delivery logs in the dashboard

Per endpoint: recent deliveries with status, attempt count, response code, response body (truncated), and next-retry time. Per event: full payload as sent. This page is where your support burden goes to die: "did you send it?" answers itself. PostHog and Stripe both make this a first-class dashboard surface.

7. Consumer documentation

The design isn't done until the consumer docs exist. Minimum: verify-the-signature code samples, the at-least-once + unordered contract, the retry schedule, ACK-fast guidance, and a local-testing story (CLI forwarder à la stripe listen, or at minimum a "send test event" button).

Artifacts to produce

Drive toward concrete output, typically in this order:

1. Design doc covering the seven sections above with the user's actual choices.
2. Envelope JSON Schema + the event type registry (coordinate with event-taxonomy).
3. DB schema for webhook_endpoints, events, deliveries (attempt rows, not overwritten status).
4. Sender implementation (or Svix integration) in the user's stack.
5. Consumer docs page with verification samples.

Related Skills

• event-taxonomy: governs the type field; keep webhook and analytics naming unified.
• api-keys: prefixing and secret-handling patterns shared by whsec_ endpoint secrets.
• billing-entitlements: webhooks are a core primitive, so never gate them by plan, gate volume instead.
• devtool-context: create the context file first if it doesn't exist.