Async patterns

Async Patterns

Synchronous = caller waits for result. Asynchronous = caller sends work and moves on. Use async when: the work is slow, the caller doesn't need the result immediately, or you need to decouple systems.

When to Go Async

Situation	Sync or Async?
User requests data, needs it now	Sync
Send confirmation email after signup	Async
Process payment (user waiting for result)	Sync (with timeout)
Generate PDF report	Async (return job ID, poll for result)
Notify 5 services about an event	Async (pub/sub)
Resize uploaded image	Async
Real-time API response	Sync
Analytics event recording	Async (fire-and-forget)

Core Patterns

1. Work Queue (Task Queue)

Producer → [Queue] → Consumer(s)

One message, one consumer. Load distributed across workers.

Use for: Background jobs, email sending, image processing, PDF generation.

Producer enqueues: { type: "send_email", to: "[email protected]", template: "welcome" }
Consumer picks up job, processes, acknowledges.
If consumer crashes → message redelivered to another consumer.

Key decisions:

• At-least-once delivery (default, safe) vs at-most-once (lossy, fast)
• Retry policy: how many times? Exponential backoff?
• Dead letter queue: where do permanently failed messages go?
• Concurrency: how many consumers? Rate limiting?

2. Pub/Sub (Fan-Out)

Publisher → [Topic] → Subscriber A
                    → Subscriber B
                    → Subscriber C

One message, multiple consumers. Each gets a copy.

Use for: Event notification, audit logging, cross-service sync.

Order service publishes: { event: "order.placed", order_id: "123", amount: 5000 }
  → Inventory service: reserves items
  → Email service: sends confirmation
  → Analytics service: records event

Key decisions:

• Subscribers are independent - one failing doesn't block others
• Message ordering: guaranteed per partition/key? Or unordered?
• Subscriber lag: what happens when a consumer falls behind?

3. Request-Reply (Async RPC)

Requester → [Request Queue] → Worker
         ← [Reply Queue]   ←

Use for: Long-running operations where the caller eventually needs the result.

1. Client POSTs /reports → returns { job_id: "abc", status: "pending" }
2. Worker processes the report
3. Client polls GET /reports/abc → { status: "completed", url: "..." }
   (or receives a webhook callback)

4. Event Sourcing

Commands → Event Store → [event1, event2, event3, ...] → Projections (read models)

Store events (facts that happened), not current state. Rebuild state by replaying events.

Use for: Audit-critical systems, financial transactions, collaborative editing.

Caution: High complexity. Only use when you genuinely need full event history and the ability to rebuild state at any point in time.

Message Design

Message Structure

{
  "id": "msg_abc123",
  "type": "order.placed",
  "version": 1,
  "timestamp": "2024-03-15T10:23:45Z",
  "source": "order-service",
  "correlation_id": "req_xyz789",
  "data": {
    "order_id": "ord_123",
    "user_id": "usr_456",
    "total_cents": 5000
  }
}

Rules:

• Every message has a unique id (for deduplication)
• Every message has a type and version (for evolution)
• Include correlation_id for tracing through async flows
• Include timestamp for ordering and debugging
• data contains the payload - keep it self-contained

Idempotency

Messages WILL be delivered more than once. Your consumer MUST handle duplicates.

Consumer receives message with id: "msg_abc123"
  → Check: have I already processed "msg_abc123"?
    YES → skip (already done)
    NO  → process, then record "msg_abc123" as processed

Store processed message IDs in a database with a TTL. Or design operations to be naturally idempotent (SET is idempotent, INCREMENT is not).

Failure Handling

Dead Letter Queue (DLQ)

Messages that fail after all retries go to a DLQ instead of being lost.

Queue → Consumer → fails → retry 1 → retry 2 → retry 3 → DLQ

DLQ rules:

• Monitor DLQ size (alert if > 0)
• Include original error in DLQ message metadata
• Build tooling to replay DLQ messages after fixing the bug
• Set retention (don't keep dead letters forever)

Poison Messages

A message that crashes the consumer every time it's processed.

Detection: same message redelivered N times
Action: move to DLQ, alert, continue processing other messages
Prevention: validate message schema before processing

Consumer Lag

When consumers can't keep up with producers.

Detection: queue depth growing over time
Actions:
  1. Scale consumers horizontally
  2. Batch process (consume N messages at once)
  3. If acceptable: shed load (drop low-priority messages)
  4. Investigate: is the consumer doing unnecessary work?

Eventual Consistency

In async systems, data across services is NOT immediately consistent. It's eventually consistent.

User places order (Order service: order = PLACED)
  → 50ms later → Inventory service: items reserved
  → 200ms later → Email service: confirmation sent
  → 2s later → Analytics: event recorded

Handling inconsistency:

• Show the user their local state immediately (optimistic UI)
• Design reads to tolerate stale data ("last updated 5s ago")
• Use compensating actions for failures (if payment fails after order created → cancel order)

Reference: reference/queue-patterns.md - Queue comparison tables, delivery guarantees, backpressure strategies, idempotency patterns.

Technology Choices

Tool	Type	Best For
Redis (Streams/Lists)	Queue	Simple queues, low latency, volatile
RabbitMQ	Queue + Pub/Sub	Flexible routing, reliable delivery
Apache Kafka	Event log	High throughput, event sourcing, replay
AWS SQS	Queue	Managed, simple, no ops
AWS SNS + SQS	Pub/Sub	Managed fan-out
NATS	Pub/Sub	Lightweight, fast, cloud-native
BullMQ / Celery / Sidekiq	Job queue	Application-level background jobs
Database (polling)	Queue	Already have a DB, low volume, don't want new infra

Decision guide:

• < 1000 msgs/sec, simple jobs? → Database polling or Redis
• Need reliable delivery, routing? → RabbitMQ
• High throughput, event replay, multiple consumers? → Kafka
• Don't want to manage infrastructure? → SQS/SNS
• Already have Redis? → BullMQ/Redis Streams

Anti-Patterns

Anti-Pattern	Problem	Fix
Async for everything	Unnecessary complexity, hard to debug	Start sync, go async only when you have a reason
No idempotency	Duplicate processing, double charges	Store processed message IDs, design idempotent operations
No DLQ	Failed messages disappear forever	Always configure a dead letter queue
Synchronous mindset	Blocking on async result, negating the benefit	Return job ID, let client poll or receive callback
Giant messages	Queue becomes a bottleneck, consumers slow	Put data in storage, put reference in message
No monitoring	Queue backs up silently	Alert on queue depth, consumer lag, DLQ size
Ordering assumptions	Messages arrive out of order, logic breaks	Design for out-of-order, or use partitioned ordering