Observability checklist

Observability checklist

The goal isn't logs — it's answering "what happened" in under five minutes at 2am. Design backward from the question.

Logs

Structured or nothing

• JSON (or logfmt), not free text.
• Every log has: ts, level, service, request_id, msg, and relevant fields.
• Fields are snake_case or camelCase — pick one per service. No mixing.
• Durations in milliseconds as numbers, not "took 2s" strings.

Levels are a contract

• error: something failed the user can't recover from. Alert-worthy.
• warn: something unexpected but handled. Review-worthy.
• info: the shape of normal traffic. Sampling OK.
• debug: developer-only. Off in prod.
• Don't invent new levels.

What to log

• Request entry (method, path, user, request_id).
• Outcome (status, duration).
• Branch decisions that matter (authz check, cache hit/miss, retry).
• Errors with the full context you need to reproduce.

What NOT to log

• Secrets, tokens, passwords, PII — scrub before logging.
• The full request body by default. Specific fields, yes. Everything, no.
• Stack traces as separate lines — include as a stack field so they stay with the event.

Traces

Span per unit of work

• HTTP request handler = 1 span.
• Outbound call = child span.
• DB query = child span (with db.statement truncated / hashed, not raw if PII risk).
• Background job step = 1 span.

Mandatory attributes

• service.name, service.version.
• For HTTP: http.method, http.route, http.status_code.
• For DB: db.system, db.operation.
• For errors: exception.type, exception.message.

Propagation

• Every outbound request carries trace context headers.
• Every background job enqueues with trace context.
• A trace that ends at the HTTP edge and restarts at the worker is useless.

Metrics

RED for services

• Rate — requests per second, per endpoint.
• Errors — error rate, per endpoint.
• Duration — p50/p95/p99, per endpoint.

USE for resources

• Utilization — % of capacity in use.
• Saturation — queue depth / wait time.
• Errors — failed operations.

Cardinality discipline

• No unbounded tags (user_id, tenant_id, url with params). They explode your TSDB bill.
• Pre-aggregate cardinal dimensions; keep labels coarse.
• http.route (the template), not http.url (the instance).

Correlation

• request_id in: logs, traces (as request.id attr), response headers, error payloads.
• trace_id in: logs (auto-injected by SDK), response headers for user bug reports.
• A user reporting "I got a 500" should give you enough to find the request — the request_id is visible and searchable.

Per-language

Node

• Pino or Winston with JSON transport.
• OpenTelemetry SDK (@opentelemetry/sdk-node).
• Prom-client for metrics.

Python

• structlog with ProcessorFormatter.
• opentelemetry-sdk + opentelemetry-instrumentation-*.
• prometheus_client.

Go

• slog (stdlib, Go 1.21+) with JSON handler.
• go.opentelemetry.io/otel + auto-instrumentation.
• prometheus/client_golang.

Rust

• tracing + tracing-subscriber with JSON formatter.
• opentelemetry-otlp.
• prometheus crate.

Anti-patterns

• "Log everything, we'll figure it out later" — the noise hides the signal you need.
• Traces without sampling — you'll drown; sample intelligently (head-based at edges, tail-based for errors).
• Metrics where logs would do — counting "user clicked button" 1M times/day is cheaper as a metric than a log.
• Logs where metrics would do — "latency: 247ms" as a log line is wasting money; it's a histogram.
• Alerts on everything — page only on symptoms that hurt users, not causes.

Smells

• A single log line has more than ~10 fields — probably two events glued together.
• You use grep to find things — you should have a field + dashboard.
• Error messages are generic ("an error occurred") — they cost you the 2am minutes.
• console.log in prod code — belongs in a logger, tagged and leveled.