Microservices architect

Microservices Architecture Skill

Core Philosophy

Distributed systems introduce complexity that's not worth paying unless you genuinely need it. Most teams reach for microservices too early. Start with a modular monolith. Add microservices only when:

• You have independent scaling needs per domain
• You need true deployment independence
• Your team is large enough to own separate services
• Your data access patterns don't require tight consistency

The monolith-first argument: A well-structured monolith is simpler, debuggable, and deployable. Graduate to microservices when pain points prove you need them—not theoretically, but in practice.

---

1. When to Use Microservices (and When NOT To)

Decision Framework

Characteristic	Monolith	Modular Monolith	Microservices
Team size	1-5 people	5-15 people	15+ people, clear domains
Scaling needs	Uniform	Mostly uniform	Heterogeneous by domain
Deployment cadence	Single	Single, modular	Independent per service
Data consistency	Strong ACID	Mostly strong	Eventual (by necessity)
Operational overhead	Low	Low	High (requires observability obsession)
Debugging difficulty	Easy	Medium	Hard (distributed tracing required)

Red flags for premature microservices:

• "We want to scale independently" (you don't, yet)
• "We want to use different technologies" (this is a smell, not a feature)
• "We need faster deployment" (deploy your monolith modules independently first)
• "Different teams own different parts" (organize code, not infrastructure)

---

2. Service Decomposition

Strategies

Domain-Driven Design (DDD)

• Identify bounded contexts in your domain
• Each context maps to one service boundary
• Within a context: tight coupling OK. Between contexts: explicit contracts only.
• Example: Order service, Inventory service, Billing service—not User service (cross-cutting).

By Business Capability

• Product Catalog, Order Management, Payments, Fulfillment
• Aligns with org structure and team ownership
• Each service owns its data and API

By Data Ownership

• Who owns this data at the business level?
• Avoid shared tables across services
• One service writes; others read via events or APIs

Anti-patterns

• Chatty services: Designing services that require 10 calls to fulfill a request. Coalesce.
• Shared databases: Couples services at the data layer. Own your data.
• God services: 10+ responsibilities. You've just moved complexity, not reduced it.

---

3. Communication Patterns

Synchronous

REST/HTTP

• Simple, debuggable, browser-friendly
• Tight coupling via request/response
• Use for immediate consistency needs (user-initiated actions)
• Timeout is your friend: always set explicit timeouts. Cascade failures otherwise.

gRPC

• Typed, contract-first (Protocol Buffers)
• Binary (faster, smaller payload)
• Streaming support
• Use when: high throughput, low latency, strong contracts matter
• Your FastAPI backend can call gRPC services via Python clients

Asynchronous

Message Queues (RabbitMQ, Kafka, AWS SQS)

• Decouple timing of request and response
• Services don't need to know about each other
• Use for: background jobs, eventual consistency, fire-and-forget workflows
• Example: Order placed → publish OrderCreated event → Inventory service consumes and reserves stock

Event-Driven Architecture

• Services emit events when state changes
• Other services subscribe and react
• Single source of truth: events, not shared state
• Kafka is the golden standard (immutable log, replay-able)

Saga Pattern (Distributed Transactions)

• For cross-service workflows that must complete or all rollback
• Choreography: services listen and trigger each other (implicit, hard to debug)
• Orchestration: dedicated saga coordinator (explicit, easier to reason about)
• Example: CreateOrder saga → Reserve inventory → Process payment → Confirm shipment (or all rollback)
• This complexity is a smell: if you need sagas, your service boundaries are probably wrong

Before/After Communication Patterns

Example 1: Synchronous Chain → Async Event-Driven

# BEFORE: Synchronous chain (Order Service calls Inventory Service calls Warehouse Service)
# FastAPI Order Handler
@app.post("/api/orders")
async def create_order(req: OrderRequest):
    order = Order(user_id=req.user_id, total=req.total)
    db.save(order)

    # Direct synchronous call to Inventory Service
    inv_response = httpx.post(
        "http://inventory-service/reserve",
        json={"order_id": order.id, "items": req.items}
    )
    if not inv_response.ok:
        db.delete(order)  # Rollback
        raise Exception("Inventory unavailable")

    # Direct synchronous call to Warehouse Service
    warehouse_response = httpx.post(
        "http://warehouse-service/schedule",
        json={"order_id": order.id}
    )
    if not warehouse_response.ok:
        # Complex rollback logic...
        pass

    return {"order_id": order.id, "status": "confirmed"}

Problems: Tight coupling, cascading failures, complex rollback, slow response time.

# AFTER: Async event-driven (Order Service publishes event, Inventory and Warehouse listen independently)
# Order Service
@app.post("/api/orders")
async def create_order(req: OrderRequest):
    order = Order(user_id=req.user_id, total=req.total, status="pending")
    db.save(order)

    # Publish event and return immediately
    event = {
        "event_type": "order.created",
        "order_id": order.id,
        "items": req.items
    }
    await kafka_producer.send("orders", json.dumps(event))

    return {"order_id": order.id, "status": "pending"}

# Inventory Service (independent consumer)
async def inventory_consumer():
    async for message in kafka_consumer.subscribe("orders"):
        event = json.loads(message.value)
        if event["event_type"] == "order.created":
            try:
                reserve_inventory(event["order_id"], event["items"])
                publish_event("inventory.reserved", order_id=event["order_id"])
            except InsufficientStock:
                publish_event("order.failed", order_id=event["order_id"], reason="no_stock")

# Warehouse Service (independent consumer)
async def warehouse_consumer():
    async for message in kafka_consumer.subscribe("orders"):
        event = json.loads(message.value)
        if event["event_type"] == "inventory.reserved":
            schedule_shipment(event["order_id"])
            publish_event("shipment.scheduled", order_id=event["order_id"])

Benefits: Services decouple, fast response (no waiting), independent scaling, easier recovery.

Example 2: Shared Database → Database-per-Service

# BEFORE: Shared database (Order and Inventory services both read/write shared tables)
# Order Service: reads inventory directly
SELECT o.id, o.total, i.quantity_available
FROM orders o
JOIN inventory i ON o.product_id = i.product_id

# Inventory Service: also manages inventory table
UPDATE inventory SET quantity_available = quantity_available - 1 WHERE product_id = ?

# Problem: Race conditions, tight coupling, cascading schema changes

# AFTER: Database-per-service (Inventory Service owns inventory data)
# Order Service: has its own schema, calls Inventory Service API
class Order(Base):
    __tablename__ = "orders"
    id = Column(Integer, primary_key=True)
    user_id = Column(Integer)
    product_id = Column(Integer)  # No foreign key to inventory
    quantity = Column(Integer)
    total = Column(Decimal)

# To check availability:
async def can_fulfill_order(product_id: int, quantity: int) -> bool:
    response = await httpx.get(
        f"http://inventory-service/products/{product_id}/availability",
        params={"quantity": quantity}
    )
    return response.json()["available"]

# Inventory Service: owns its database and API
class InventoryStock(Base):
    __tablename__ = "stock"
    product_id = Column(Integer, primary_key=True)
    quantity_available = Column(Integer)
    last_updated = Column(DateTime)

@app.get("/products/{product_id}/availability")
async def check_availability(product_id: int, quantity: int):
    stock = db.query(InventoryStock).filter(
        InventoryStock.product_id == product_id
    ).first()
    return {
        "product_id": product_id,
        "available": stock.quantity_available >= quantity if stock else False
    }

Benefits: Scaling independent, schema evolution independent, clear data ownership.

API Gateway

• Single entry point for external clients
• Handles routing, authentication, rate limiting, response aggregation
• Anti-pattern: Making the gateway smart. Keep it dumb (routing + cross-cutting concerns only).

---

4. Data Management

Database-per-Service Pattern

Rule: Each service owns one database. No cross-service queries.

Why?

• Service independence: deploy without coordinating migrations
• Scaling: optimize schema per service needs
• Failure isolation: one DB goes down, others unaffected

Implementation:

• Order Service has orders DB with orders, order_items tables
• Inventory Service has separate inventory DB with stock, reservations tables
• They communicate via APIs/events, not shared tables

Eventual Consistency

The harsh reality of microservices: you cannot have strong consistency across service boundaries without synchronous blocking calls (which re-couple services).

Accept eventual consistency. Design for it:

• Order placed → Inventory increments eventually
• Display "processing" state to user during consistency window
• Implement conflict resolution (e.g., if inventory insufficient, trigger compensation)

When you need strong consistency across services: You drew the boundary wrong. Move those domains into the same service.

Event Sourcing (Use Sparingly)

Store the immutable event log instead of current state. Reconstruct state by replaying events.

Worth the complexity only if:

• You need audit trails (regulatory requirement)
• You need to debug "how did we get here?" (time-travel debugging)
• You're processing high-velocity events

Cost: Schema migrations become event migrations. Snapshots needed to avoid replaying years of events.

For your Python/FastAPI apps, consider Kafka as an event store before building custom event sourcing.

Shared Data Anti-patterns

• Shared database tables (couples services at data layer)
• "Read-only" cross-service queries (still couples; use events instead)
• Shared reference tables (duplicate them or use a reference service API)

---

5. Resilience Patterns

Every network call can fail. Design for it.

Circuit Breaker

• Track failures to a dependency
• After N failures: stop calling, return cached/default response (open state)
• Periodically retry (half-open state)
• Restore when succeeds (closed state)
• Library: pybreaker for Python

Bulkhead Pattern

• Isolate resources (threads, connections) per dependency
• If Service B is slow, it doesn't starve Service C
• Thread pools, connection pools with limits

Retry with Exponential Backoff

• Transient failures (network hiccup, timeout)
• Don't retry: 4xx errors, authentication failures
• Retry: 5xx, timeouts, connection errors
• Backoff: 100ms → 200ms → 400ms (with jitter to avoid thundering herd)

Timeout Cascades

• If you set timeout T at the gateway, and each service has timeout T, requests fail silently
• Rule: Each downstream call must have shorter timeout than the upstream caller
• Example: Client timeout 30s → Gateway timeout 28s → Service A timeout 25s → Service B timeout 20s

Graceful Degradation

• Service down? Return partial response, cached data, or feature flag off
• Don't hard-fail the entire request

---

5.5 Service Architecture Health Score

When reviewing an existing microservices system, assess health across these dimensions:

Service Architecture Health Score
==================================

Service Independence   ████░░░░░░  [Needs Work] - services have minimal coupling
Data Ownership        ███░░░░░░░  [Critical Gap] - each service owns its data
Resilience Patterns   ██████░░░░  [Good] - circuit breakers, retries, timeouts
Observability         █████░░░░░  [Solid] - tracing, logging, metrics
Deployment Independence ██░░░░░░░░ [Critical Gap] - can deploy one service without others

Overall: [Sentence verdict. E.g., "Services are well-designed but lack observability—distributed tracing and metrics are mandatory before scaling further."]

What to Check for Each Dimension

Service Independence — Are services loosely coupled?

• No cross-service database queries (couples data layer)
• No synchronous chains > 2 hops (one service → at most 2 others)
• No shared code libraries (except base contracts/shared types)
• Service A can be down without cascading failure

Data Ownership — Is data ownership clear?

• Each service has exactly one database (no shared tables)
• One service is authoritative source for each entity type
• No cross-service foreign keys (use event-driven or APIs instead)
• Read-heavy services replicate via events, not sync queries

Resilience Patterns — Can the system degrade gracefully?

• Circuit breakers on all synchronous calls (detect failing services)
• Timeouts set everywhere (prevent cascading timeouts)
• Retry logic with exponential backoff (transient failures)
• Bulkhead isolation (if Service B slow, doesn't starve Service C)
• Graceful degradation (service down? Return partial response, not error)

Observability — Can you debug a production incident in 30 minutes?

• Distributed tracing (trace single request across all services)
• Centralized logging with trace ID correlation
• Metrics on request rate, latency, error rate per service
• Circuit breaker state visible (is anything open?)
• Can you search: "trace where service=payment and latency > 5s"?

Deployment Independence — Can you push one service without deploying others?

• Service version changes don't require recompiling other services
• API contracts versioned (v1, v2, backward compatible)
• Database migrations per-service (no cross-service schema coordination)
• Different deployment cadences per service are supported

Quick Assessment Template

Use this when reviewing a microservices design:

Service Boundaries Assessment
=============================

Services Identified:
- Order Service (owns orders, order_items tables) ✓
- Inventory Service (owns stock, reservations tables) ✓
- Payment Service (owns payment_transactions table) ✓

Cross-Service Calls:
1. Order Service → Inventory Service: /products/{id}/availability
   [Sync or Async?] Async (event-driven)
   [Timeout?] N/A for async

2. Order Service → Payment Service: /charges
   [Sync or Async?] Sync (immediate confirmation needed)
   [Timeout?] Yes, 10s, with retry (max 2x)

Data Ownership:
- Orders ← Order Service (authoritative)
- Inventory Stock ← Inventory Service (authoritative)
- Inventory reads in Order Service ← via API calls (not sync query)

Issues Found:
- CRITICAL: Order Service queries payment_transactions table directly (should call Payment API instead)
- HIGH: No circuit breaker on Order → Inventory call (payment_service down crashes orders)
- MEDIUM: Async events published but no dead letter queue (messages silently lost on error)

Recommendation:
Extract direct database query to API call. Add circuit breaker. Implement DLQ for reliability.

---

6. Service Mesh & Observability

Observability is Non-Negotiable

You cannot run microservices without:

Distributed Tracing (Jaeger, Zipkin)

• Trace a single request across all services
• Identify which service is slow
• Shows network latency, service processing time, queueing

Centralized Logging (ELK, Datadog, Cloud Logging)

• Correlate logs across services via trace ID
• Search "all logs where service=payment-service AND status=error"

Metrics (Prometheus, Datadog)

• Request rate, latency, error rate per service
• Circuit breaker state, queue depth, database connections

Service Discovery

• Services move (deployments, scaling, failures)
• You need: DNS or a registry (Consul, Kubernetes DNS)
• Client-side (smart client) vs server-side (proxy) discovery
• Kubernetes does this for you

Health Checks & Readiness Probes

• Liveness: Is the service alive? If not, restart it.
• Readiness: Is it ready to handle traffic? (DB migrations done, connections open?)
• Load balancers check readiness before routing requests

---

7. Migration Patterns

Strangler Fig Pattern (Monolith → Microservices)

• Don't rewrite. Extract incrementally.
• New feature → new service
• Old feature → routed through proxy to monolith until replaced
• Gradually strangle the monolith

Example: Monolith handles Orders. Extract Order Service:

1. Deploy Order Service alongside monolith
2. Proxy wraps monolith requests: if /orders/* → call Order Service, else → call monolith
3. Migrate data, test thoroughly
4. Kill monolith routing for /orders/*
5. Repeat for Inventory, Payments, etc.

Anti-Corruption Layer

• Monolith and Service speak different languages (schema, terminology)
• Adapter layer translates between them
• Prevents service from knowing monolith internals

Parallel Run / Dark Launching

• New service runs alongside old, processing same requests
• Compare results, don't return new service response to users yet
• Verify correctness before switching traffic

---

Assessment Framework (for Reviews)

Service Boundaries

• Does each service own its data?
• Is there a clear business reason for this boundary?
• Count the cross-service calls per request (should be < 3)

Coupling Analysis

• Shared database tables? (Bad)
• Synchronous chains? (Reconsider boundary)
• Hard dependencies on request/response? (Are they in the same service?)

Communication Evaluation

• Is REST used for eventual consistency? (Reconsider—use async)
• Is async used for immediate consistency? (Reconsider—use sync or move boundary)
• Are timeouts set everywhere? (Non-negotiable)

Data Consistency Review

• How do you handle "Saga failed halfway"?
• Do you have compensating transactions?
• Is eventual consistency acceptable here?

Failure Modes

• What happens if Service B is down? (Degraded gracefully?)
• What happens if the message queue is full? (Backpressure?)
• What happens if a network partition splits your services?

---

Deliverables

Design Consultations

• ADR (Architecture Decision Record): Service boundaries, communication style, data strategy
• Service boundary diagram (text): Show services, their APIs, and data ownership
• API contracts: OpenAPI/gRPC schemas for service-to-service calls
• Data flow diagram: Show synchronous calls, events, and query paths
• Failure mode analysis: FMEA for critical paths (payment, order creation)

Code Reviews

• Assess service boundaries and inter-service coupling
• Evaluate resilience (timeouts, retries, circuit breakers)
• Check observability (tracing, logging, metrics)
• Validate data consistency strategy
• Point out premature optimization (microservices where monolith sufficient)

---

Remember

• Every service boundary is a network call. Network calls fail.
• Microservices don't solve organizational problems (communication, unclear responsibilities).
• Eventual consistency is the price of loose coupling.
• If you can't draw why a boundary exists, it's wrong.
• Observability isn't optional—it's survival.