Observability

Observability (Instrumentation + Verification)

Overview

Make observability consistent and actionable: every boundary emits traces, metrics, and structured logs that correlate via IDs and stable fields.

This is intentionally opinionated: you should be able to answer “what happened?” with log → trace → metrics within a minute.

Inputs / Outputs

Inputs: Boundary code to instrument; service name; existing instrumentation (if any); correlation IDs in use. Outputs: Instrumented code (traces, metrics, structured logs); field contract (stable keys); verification commands. Consumed by debug (for triage) and finish (for spot-check).

Workflow

1. Name the decision this telemetry will inform (for example: keep current architecture, tune a timeout, roll out a migration).
2. Define the unit of work (one trace): HTTP request, gRPC call, job run, queue message, WebSocket action.
3. Define the measurement ladder:
   • 3 leading indicators (move within days)
   • 3 lagging outcomes (move within weeks/months)
   • owner + review cadence + trigger for action
4. Instrument end-to-end:
   • traces: spans around the unit of work + key downstream calls
   • metrics: RED for the boundary + a few domain metrics
   • logs: structured JSON that includes correlation IDs
5. Declare the field contract (stable keys).
6. Add guardrails (PII rules, label cardinality rules, sampling/log levels).
7. Verify correlation in a failure case (error log includes traceId; trace contains downstream spans; metrics show error rate).
8. Define the operating ritual:
   • where metrics/logs/traces are reviewed
   • who decides and who executes follow-up
   • what threshold triggers rollback/escalation

Minimum viable execution

When context or time is constrained, these are the load-bearing steps:

1. Name the decision (step 1) — every metric needs a named decision it informs.
2. Instrument RED at boundaries (step 4) — request count, error count, duration histogram per boundary.
3. Add trace ID correlation to logs (step 4-5) — structured logs must include traceId for log → trace linkage.
4. Verify correlation in a failure case (step 7) — error log includes traceId, trace shows downstream spans, metrics reflect the error.

Steps that can be cut under pressure: measurement ladder (step 3), domain-specific metrics, operating ritual (step 8).

Clarifying Questions

• What decision does this telemetry inform (keep current architecture, tune a timeout, roll out a migration, etc.)?
• What boundaries need instrumentation (HTTP handlers, gRPC methods, DB clients, consumers, jobs, WebSockets)?
• Is there existing telemetry infrastructure (OpenTelemetry, Prometheus, Grafana, Datadog, etc.)?
• What correlation IDs already exist in the system (traceId, requestId, sessionId)?
• Are there PII/privacy constraints on what can be logged or labeled?
• Who will review these metrics, and how often (owner + cadence)?

Chooser (What To Instrument)

Start with the user-impact boundaries:

• HTTP handlers: one root span per request + RED metrics per route template.
• gRPC methods: one root span per RPC + RED metrics per service/method.
• DB/cache clients: child spans per query/command; include target system and operation.
• Async jobs / schedulers: one root span per run; metrics for runs/success/failure/duration.
• Event consumers: one root span per message (or per batch); include message type and dedupe/idempotency metadata.
• WebSockets: session context + per-action spans; metrics for connections, messages, disconnect reasons.

Field Contract (Opinionated Defaults)

Logs (structured JSON)

Include these keys where applicable:

• service: stable service/app identifier
• env: environment (local/dev/staging/prod)
• traceId, spanId: correlation IDs (when tracing exists)
• requestId: if you use a separate request ID (often equals traceId)
• op: operation name (route template, RPC method, job name)
• userId / actorId: only if policy allows; never as a metric label
• durationMs: for timing logs (prefer metrics for aggregates)
• err: structured error (type/code, message, stack for unknown failures)

Spans (traces)

• Name spans by operation (HTTP GET /api/foo, grpc PlayerService/GetProfile, redis GET gateway:...).
• Set attributes for routing and outcome (status code, error code, retry count).
• Prefer stable, low-cardinality attributes; avoid raw request bodies.

Metrics (RED + domain)

• RED for each boundary (per route/RPC): request count, error count, duration histogram.
• Add a few domain metrics that align with product intent (tables created, orders completed, etc.).
• Avoid high-cardinality labels (no userId, no unbounded IDs); use logs/traces for per-entity detail.

Guardrails (Prevent “Telemetry Debt”)

• Cardinality discipline: metric label values must be bounded sets; default to route templates, not raw URLs.
• PII discipline: never log secrets; be explicit about what IDs are safe to log.
• Log once: avoid logging the same error in every layer; log at the boundary with enough context.
• Sample intentionally: if you sample traces, keep error traces at higher priority.
• Always end spans: long-running work should have explicit shutdown and cancellation semantics.
• Decision linkage: no metric without a named decision and action threshold.

Common failure modes

• Adds high-cardinality labels to metrics (user IDs, request bodies, unbounded strings) — causes metric explosion and storage/query issues.
• Instruments everything instead of focusing on RED metrics at boundaries — creates noise that makes real signals harder to find.
• Missing trace ID correlation — logs, traces, and metrics exist independently with no way to connect them for a single request.
• Defines metrics without a named decision, owner, or action threshold — the metric exists but nobody knows what to do when it changes.

Minimal TypeScript Snippet (Trace IDs in Logs)

If you use OpenTelemetry, you can enrich logs with the active span context:

import { context, trace } from '@opentelemetry/api';

export function getTraceLogFields(): { traceId?: string; spanId?: string } {
  const span = trace.getSpan(context.active());
  if (!span) return {};
  const { traceId, spanId } = span.spanContext();
  return { traceId, spanId };
}

Testing / Verification

• Exercise a failing request and verify:
  • the error log includes traceId
  • the trace contains downstream span(s)
  • boundary RED metrics reflect the error
• Prefer consumer-visible tests for behavior; treat telemetry verification as a local/dev smoke check unless the project already has telemetry assertions.

References

• Deeper checklists: references/checklists.md
• TypeScript instrumentation snippets: references/snippets/typescript.md
• Related patterns: Application metrics, Log aggregation, Distributed tracing, Health Check API, Audit logging, Exception tracking, Log deployments and changes
• Boundary tests: testing
• Typed errors + explicit lifetimes: typescript

Output Template

When applying this skill, return:

• The decision being informed and the measurement ladder (leading/lagging, owner, cadence, trigger).
• The instrumentation plan (which boundaries, what telemetry, what fields).
• The minimal code changes (where to start spans, where to log, what metrics to add).
• The verification steps (how to reproduce and correlate log → trace → metrics).