Logging best practices

Logging Best Practices

Operating Model

Treat logs as queryable production facts, not a stream of developer diary entries.

Prefer one wide event per request or service hop. Build that event throughout the request lifecycle, attach request, service, user, business, dependency, timing, and outcome context, then emit it once when the final status is known.

Structured logging is required but not sufficient. JSON with inconsistent keys, too few fields, or no business context still produces weak telemetry.

Use OpenTelemetry as a collection, propagation, and export layer when it fits the codebase. Do not treat OpenTelemetry as a substitute for deliberate instrumentation; it will not infer product-specific context by itself.

Workflow

1. Start from the debugging questions operators must answer, such as which customers were affected, which rollout was active, which dependency failed, or which business object was involved.
2. Identify the event boundary: HTTP request, RPC call, background job, queue message, cron run, CLI operation, or other unit of work.
3. Design the event schema before adding code. Read wide-event-field-guide.md when choosing fields, naming, sampling policy, or migration steps.
4. Implement a request-scoped event builder or context object. Initialize it at the boundary with correlation, service, deployment, route, and start-time fields.
5. Enrich the event in handlers and service calls with business context as it becomes available. Prefer typed helpers or schema validation where the language and codebase support them.
6. On failure, capture typed, sanitized error fields and preserve normal error propagation. Do not swallow or transform exceptions just to make logging easier.
7. In a finally/defer/after-hook path, compute duration, set outcome fields, apply tail sampling, and emit the event once.
8. Validate the implementation by writing the concrete queries the new events should support. If the query cannot be answered directly from event fields, the schema is not complete.

Implementation Standards

Keep field names stable and consistent across services. Follow the repository's existing casing and telemetry conventions when they are coherent; otherwise prefer clear snake_case names for event fields.

Capture high-cardinality identifiers when they are needed for debugging, such as user, tenant, request, order, cart, job, message, or external-provider IDs. Handle privacy deliberately: redact secrets, tokens, credentials, payment data, unrestricted payloads, and raw personal data unless the codebase has an explicit safe policy for that field.

Record both low-cardinality fields for aggregation and high-cardinality fields for investigation. Useful production telemetry needs both. Do not drop high-cardinality identifiers over cost objections: modern columnar log backends query them efficiently, and volume is controlled through tail sampling, not by removing debugging context.

Represent values with queryable types: numbers as numbers, booleans as booleans, timestamps as timestamps, durations in explicit units such as _ms, and money in smallest currency units plus currency.

Keep traces, metrics, and wide events aligned. Reuse trace/request IDs, enrich spans with the same context when appropriate, and derive dashboards from the same events instead of maintaining a separate low-context logging path.

Centralize logger/exporter configuration at application startup so formatting, environment fields, redaction, sampling metadata, and destination behavior stay consistent across modules.

Avoid replacing a wide event with scattered strings such as one log per code step. Temporary debug logs may be acceptable during active development, but they must not be the only durable signal for production behavior.

Tail Sampling

Make sampling decisions after the unit of work completes, when outcome and latency are known. Always retain high-signal events such as errors and slow requests; randomly sample the remaining happy-path traffic at a documented rate (1-5% is a common starting point). Read the tail sampling policy in wide-event-field-guide.md for the full retention rules and sampling metadata fields.

Review Checklist

• Can an operator answer the target incident questions with direct structured queries instead of grep?
• Is there exactly one authoritative event per boundary per service hop?
• Are request, trace, service, deployment, environment, route, duration, status, and outcome fields present?
• Are user, tenant, business object, feature flag, dependency, and error fields present when relevant?
• Are error fields typed, sanitized, and specific enough to group by root cause?
• Are correlation identifiers propagated across service calls, queue messages, jobs, and downstream dependencies?
• Is logger/exporter configuration centralized instead of redefined ad hoc in unrelated modules?
• Are high-cardinality fields retained where they matter, with privacy controls applied?
• Does tail sampling retain the events most likely to explain incidents?
• Are tests or verification artifacts present for success, failure, and sampling behavior?

Source

This skill distills the logging guidance from Boris Tane's article "Logging sucks." at https://loggingsucks.com/.