Standards observability

Observability Standards

Provides: Structured logging patterns, metrics design, error tracking, performance monitoring, and debugging strategies.

Quick Reference

Core Philosophy: Observable, Debuggable, Measurable
Golden Rule: If you can't see it in production, you can't fix it

Critical Patterns (use these):

• ✅ Structured logging with bounded, non-sensitive context
• ✅ Meaningful error messages with safe diagnostic details
• ✅ Business SLIs and SLOs tied to user outcomes
• ✅ Low-cardinality metric labels and normalized route names
• ✅ OpenTelemetry or W3C trace context across service boundaries
• ✅ Alerts on user-impacting symptoms, each with an owner and runbook

Anti-Patterns (avoid these):

• ❌ Silent failures (no logs, metrics, or traces)
• ❌ Generic error messages ("Error" or "Failed")
• ❌ Raw PII, secrets, tokens, or payloads in logs
• ❌ Logging everything (noise hides signals)
• ❌ High-cardinality labels (raw URLs, IDs, emails)
• ❌ Shared mutable logger context across concurrent requests

---

Core Philosophy

Observable: Every significant action is logged and measurable
Debuggable: Logs contain enough context to diagnose issues
Measurable: Metrics track what matters for users and operations

Logging Standards

Structured Logging

✅ DO: Use structured (JSON) logging

// ✅ Good - Structured, contextual
logger.info({
  event: 'user_created',
  userId: user.id,
  email: user.email,
  timestamp: new Date().toISOString(),
  duration: performance.now(),
  environment: process.env.NODE_ENV
});

// ✅ Good - Template with context
logger.info('User created', {
  userId: user.id,
  email: user.email,
  source: 'signup_form'
});

❌ DON'T: Use free-form text logging

// ❌ Bad - No context, unstructured
console.log('Created user');

// ❌ Bad - Unstructured concatenation
logger.log('User ' + user.id + ' was created at ' + new Date());

// ❌ Bad - No actionable information
logger.error('Something went wrong');

Log Levels

Use appropriate log levels to control noise:

• ERROR: Actionable errors requiring investigation or manual intervention

  logger.error('Database connection failed', {
    database: 'users-db',
    error: error.message,
    retry: 3,
    nextRetry: '5s'
  });
  

• WARN: Degraded performance, approaching limits, recoverable issues

  logger.warn('Cache miss rate high', {
    hitRate: 0.45,
    threshold: 0.8,
    action: 'check_cache_configuration'
  });
  

• INFO: Important business events (user actions, deployments, config changes)

  logger.info('Deployment started', {
    version: '1.2.3',
    environment: 'production',
    deployedBy: 'ci/cd'
  });
  

• DEBUG: Detailed information for troubleshooting (not in production by default)

  logger.debug('Processing payment', {
    orderId: '12345',
    amount: 99.99,
    processor: 'stripe'
  });
  

Log Context & Correlation

✅ DO: Include correlation IDs for request tracing

// Middleware: attach a request-scoped child logger
function requestContextMiddleware(req, res, next) {
  const inboundRequestId = req.headers['x-request-id'];
  const requestId = isValidRequestId(inboundRequestId)
    ? inboundRequestId
    : generateId();

  req.log = logger.child({ requestId, userId: req.user?.id });
  res.setHeader('x-request-id', requestId);
  next();
}

// In application code - requestId is included by the child logger
req.log.info('Processing order', { orderId: '123' });
// Output includes: { requestId: 'abc123', userId: 'user456', orderId: '123' }

Do not store request context in shared mutable logger state. Use child loggers, AsyncLocalStorage, or OpenTelemetry context propagation, so concurrent requests cannot leak userId, requestId, or trace context into each other's logs.

Validate an inbound correlation ID before echoing or storing it. An unvalidated header is both a log-injection vector and an unbounded cardinality source.

✅ DO: Include relevant context for debugging

logger.error('Payment processing failed', {
  orderId: order.id,
  customerId: customer.id,
  amount: order.total,
  paymentGateway: 'stripe',
  errorCode: error.code,
  errorMessage: sanitize(error.message),
  retryable: error.retryable,
  attempt: attempt,
  maxAttempts: MAX_RETRIES
});

What to Log

✅ DO log:

• User actions (login, signup, purchase, upload)
• State transitions (order created → paid → shipped)
• Errors with context (not just the error, but what was being done)
• Performance thresholds (slow queries, timeouts)
• Configuration changes and deployments
• Security events (auth failures, permission denials)

❌ DON'T log:

• Sensitive data (passwords, API keys, tokens, PII without redaction)
• Full request or response payloads unless explicitly sampled and redacted
• Internal implementation details (loop counters, temp variables)
• Every function call (use DEBUG level, disable in production)
• Duplicate information (already in metrics)
• Unbounded values that create noisy searches and retention problems

---

Error Tracking Standards

Structured Error Messages

✅ DO: Provide actionable error context

// ✅ Good - Clear, actionable, contextual
class PaymentError extends Error {
  constructor(message, context) {
    super(message);
    this.name = 'PaymentError';
    this.code = context.code;
    this.orderId = context.orderId;
    this.retryable = context.retryable;
    this.timestamp = new Date();
  }
}

throw new PaymentError(
  'Card declined: Insufficient funds',
  {
    code: 'card_declined',
    orderId: '12345',
    retryable: true
  }
);

❌ DON'T: Use vague error messages

// ❌ Bad - No context
throw new Error('Failed');

// ❌ Bad - Implementation detail, not user action
throw new Error('JSON.parse failed');

// ❌ Bad - No code or recovery info
throw new Error('Database error');

Error Handling Pattern

// ✅ Good - Clear separation of concerns
async function processPayment(order) {
  try {
    const result = await paymentGateway.charge(order);
    
    logger.info('Payment successful', {
      orderId: order.id,
      amount: order.total,
      transactionId: result.id
    });
    
    return { success: true, transactionId: result.id };
    
  } catch (error) {
    // Categorize error for monitoring
    const isRetryable = error.code === 'network_error' || error.code === 'timeout';
    const isFatal = error.code === 'card_declined' || error.code === 'invalid_card';
    
    logger.error('Payment failed', {
      orderId: order.id,
      amount: order.total,
      errorCode: error.code,
      errorMessage: error.message,
      isRetryable,
      isFatal,
      stack: error.stack
    });
    
    // Return structured error instead of throwing
    return {
      success: false,
      code: error.code,
      message: error.message,
      retryable: isRetryable,
      fatal: isFatal
    };
  }
}

---

Metrics Standards

Business Metrics (What Matters)

Track metrics that indicate user value and business health:

✅ DO: Track business outcomes

// User engagement
metrics.increment('users.signup', 1);
metrics.increment('users.login', 1);
metrics.increment('orders.created', 1);
metrics.gauge('active_users', activeUserCount);

// Conversion metrics
metrics.gauge('conversion_rate', (purchased / visited) * 100);

// Quality metrics
metrics.gauge('error_rate', (errors / requests) * 100);
metrics.gauge('payment_success_rate', (succeeded / total) * 100);

❌ DON'T: Track only technical metrics

// ❌ Not actionable
metrics.increment('function_calls');
metrics.increment('loop_iterations');

// ❌ Too granular
metrics.increment('variable_assignments');

Performance Metrics

✅ DO: Measure operations that affect users

// ✅ Good - Meaningful performance metrics
const timer = metrics.startTimer('database.query.duration');
const results = await db.query('SELECT * FROM users WHERE active = true');
timer.end({ operation: 'select_active_users' });

// ✅ Good - API response times by endpoint
const apiTimer = metrics.startTimer('api.request.duration');
await handleRequest();
apiTimer.end({ method: req.method, path: req.path, status: res.status });

// ✅ Good - Queue depth and processing time
metrics.gauge('job_queue.depth', queue.length);
const jobTimer = metrics.startTimer('job.processing.duration');
await processJob(job);
jobTimer.end({ jobType: job.type });

Metric Guidelines

• Counter: Monotonic cumulative values (requests, errors, conversions)
• Gauge: Current value (queue length, memory, active connections)
• Histogram: Distribution (response times, payload sizes)
• Summary: Client-side quantiles when the backend cannot aggregate histograms

Keep label values low-cardinality and bounded. Use route templates, status classes, error codes, and enum-like values. Do not tag metrics with raw URLs, emails, UUIDs, payload values, stack traces, or unconstrained user input. Every distinct label combination is a separate time series, so one unbounded label can multiply storage cost and make queries unusable.

Pattern:

// ✅ Good - Clear metric names with tags
metrics.timing('api.request', duration, {
  method: 'POST',
  route: '/api/users/:id', // route template, not the resolved path
  status: 201
});

metrics.gauge('cache.memory', bytes, {
  cache: 'user_sessions'
});

metrics.increment('errors', 1, {
  type: 'validation_error',
  field: 'email'
});

---

Distributed Tracing

Request Correlation

✅ DO: Prefer OpenTelemetry and W3C Trace Context

Use framework auto-instrumentation first. When manual propagation is needed, inject and extract the W3C traceparent and tracestate headers rather than inventing custom trace headers. Custom headers do not interoperate with other services, proxies, or vendor backends, so a trace stops at the first boundary that does not recognize them.

// 1. Entry point: extract W3C trace context with OpenTelemetry
app.use(otelHttpMiddleware());

// 2. Service calls: propagate the active trace context
async function callUserService(userId) {
  const headers = {};
  otel.propagation.inject(otel.context.active(), headers);

  const result = await fetch('http://user-service/users/' + userId, {
    headers
  });
  return result;
}

// 3. Logs include trace context from the active span
const span = otel.trace.getActiveSpan();
logger.info('User retrieved', {
  traceId: span?.spanContext().traceId,
  spanId: span?.spanContext().spanId,
  userId,
  durationMs
});

If legacy systems require custom x-request-id or x-trace-id headers, validate their length and format before echoing or storing them, and regenerate invalid values. This avoids both log injection and cardinality blowups.

---

Performance Profiling

Profiling Guidance

• Use distributed traces to find the slow span before adding logs.
• Capture CPU, heap, and allocation profiles for reproducible hotspots.
• Prefer sampling profiles in production over always-on verbose diagnostics.
• Correlate profiles with deploy versions, feature flags, and traffic shape.
• Redact heap dumps and query samples before sharing or storing them.

Checklist: Production Observability

• All errors logged with context (not silent failures)
• Business metrics tracked (not just technical metrics)
• Request correlation/tracing (can follow requests across services)
• Performance thresholds monitored (slow queries, timeouts)
• Log levels appropriate (no over-logging in production)
• Sensitive data redacted from logs (PII, secrets, tokens, payloads)
• Request context is per-request, not shared mutable logger state
• Error messages actionable (help debugging/fixing)
• Metrics have meaningful names and bounded, low-cardinality labels
• Trace context propagates with W3C headers across every service boundary
• Alerts have a symptom, window, owner, and runbook
• Dashboards show user-visible metrics
• Instrumentation is reviewed after incidents and adjusted to what was missing

Debugging Checklist

• Logs contain request IDs for correlation
• Error stack traces preserved (not swallowed)
• Contextual data included (what was being done when error occurred)
• Log level appropriate to severity
• Metrics show anomalies (unusual patterns)
• Performance profiling tools accessible (flame graphs, traces)
• Database query logs available (with execution time)

---

Alerts & Monitoring

Alert Guidelines

Every alert needs a user-impacting symptom, a threshold, a time window, an owner, and a runbook. Prefer SLO burn-rate alerts and symptom-based conditions over raw component metrics.

✅ DO: Alert on problems, not noise

// ✅ Good - Alert on user-impacting issues
{
  name: 'High 5xx Error Rate',
  condition: '5xx_error_rate > 0.05 for 5m',
  severity: 'critical',
  owner: 'platform-oncall',
  action: 'page_oncall',
  runbook: 'https://runbooks.example.com/api-5xx',
  dashboard: 'https://dashboards.example.com/api-health'
}

{
  name: 'Payment Processing Slow',
  condition: 'payment_processing_time_p99 > 5000ms for 10m',
  severity: 'warning',
  owner: 'payments-team',
  action: 'notify_team',
  runbook: 'https://runbooks.example.com/payment-latency'
}

// ✅ Good - Alert on SLO burn rate
{
  name: 'Availability SLO Burn Rate',
  condition: 'availability_error_budget_burn_rate > 14 for 10m',
  severity: 'critical',
  owner: 'service-owner',
  action: 'page_oncall',
  runbook: 'https://runbooks.example.com/slo-burn-rate'
}

❌ DON'T: Alert on every metric

// ❌ Bad - Creates alert fatigue
{
  condition: 'requests_total > 0',
  action: 'page_oncall'
}

// ❌ Bad - Not actionable
{
  condition: 'some_metric_changed',
  action: 'notify_team'
}

---

Example: Complete Observability Integration

Read the complete integration example when a task needs an end-to-end implementation model.

See also

• Load role-site-reliability-engineer for SLO targets, alert review, and production readiness.
• Load workflow-debugging when investigating a live failure or regression.
• Load standards-security for redaction requirements around PII and secrets.