Go observability

Go Observability — Metrics, Traces, and Correlation

Production Go services need at least two always-on signals to be debuggable: metrics (aggregated measurements for alerting and SLOs) and traces (per-request flow showing where time went). The third deliverable is correlation: a P99 metric spike must lead, in one click, to the trace that caused it.

Logging is not in this skill. Structured logging with log/slog, log levels, and zap/logrus/zerolog migration belong to the go-logging skill. This skill only references logs in the context of correlating them with traces.

Core Rules

1. A feature is not done until it is observable. New code MUST export at least: one Counter for operations, one Counter for errors, one Histogram for latency.
2. Histograms, not Summaries, for latency. Summaries cannot be aggregated across instances; Histograms support histogram_quantile() server-side.
3. Label cardinality is bounded. Never put unbounded values (user IDs, full URLs, request IDs) in Prometheus labels. Use route patterns, status classes, method.
4. Context flows everywhere. A function that does I/O takes ctx context.Context as its first argument. No context = no trace propagation = no correlation.
5. Record errors on the span. When a span ends in failure, call span.RecordError(err) and span.SetStatus(codes.Error, ...). A green span hides a real failure.
6. Correlate or it didn't happen. Inject trace_id into logs, attach exemplars to histograms. Otherwise the three signals are three different products.

Signal Decision

Pick the signal that matches the question. Do not log what should be a metric.

Question	Signal	Tool
How often does X happen? Error rate? Rate-per-second?	Metric (Counter)	Prometheus
What is the P99 latency of endpoint /orders?	Metric (Histogram)	Prometheus + histogram_quantile
Where did this one slow request spend its time?	Trace	OpenTelemetry
Why does latency spike at 14:32?	Metric → exemplar → trace	Prometheus + OTel + exemplars
What concrete error message did this request hit?	Log (see go-logging)	log/slog

Read references/metrics.md for Counter/Gauge/Histogram patterns, naming, and PromQL-as-comments. Read references/tracing.md for TracerProvider setup, span attributes, and otelhttp middleware.

Metrics — the 60-Second Setup

import "github.com/prometheus/client_golang/prometheus"

// rate(http_requests_total{code=~"5.."}[5m]) / rate(http_requests_total[5m])
var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method, route, status class.",
    },
    []string{"method", "route", "code"}, // ALL bounded
)

// histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
var httpLatency = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency in seconds.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route"},
)

The comment above each metric is the PromQL it is designed to answer. This makes the metric discoverable and grep-able from a dashboard.

Traces — the 60-Second Setup

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/codes"
)

func (s *OrderService) Create(ctx context.Context, in CreateOrderInput) (*Order, error) {
    ctx, span := otel.Tracer("order-service").Start(ctx, "OrderService.Create")
    defer span.End()
    span.SetAttributes(attribute.String("order.user_id", in.UserID))

    order, err := s.repo.Insert(ctx, in)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "insert failed")
        return nil, fmt.Errorf("creating order: %w", err)
    }
    return order, nil
}

Every service method, every DB query, every external HTTP call gets a span. Context must flow into s.repo.Insert(ctx, ...) so the DB span is a child of the service span.

Correlation

Metrics → Traces with Exemplars

An exemplar attaches a single trace_id to a histogram observation. In Grafana, click the dot on a P99 spike and you land on the trace.

obs := httpLatency.WithLabelValues(r.Method, routePattern)
sc := trace.SpanContextFromContext(ctx)
if eo, ok := obs.(prometheus.ExemplarObserver); ok && sc.IsValid() {
    eo.ObserveWithExemplar(elapsed.Seconds(),
        prometheus.Labels{"trace_id": sc.TraceID().String()})
} else {
    obs.Observe(elapsed.Seconds())
}

Logs → Traces

Use the otelslog bridge (see go-logging skill) so every slog.InfoContext(ctx, ...) call automatically emits trace_id and span_id. You can then grep logs by trace_id when starting from a trace, or jump from a log line to the trace.

Read references/correlation.md for exemplar wiring details, request-id propagation, and the end-to-end "metric spike → trace → log" workflow.

Context Propagation

// Bad — breaks trace propagation; the DB call starts a new root trace.
result, err := db.Query("SELECT ...")

// Good — the DB span is a child of the HTTP request span.
result, err := db.QueryContext(ctx, "SELECT ...")

Every I/O boundary takes ctx: HTTP client, database, gRPC client, message queue. Use otelhttp.NewTransport to instrument outbound HTTP and otelhttp.NewHandler for inbound.

Anti-Patterns

Anti-pattern	Why it hurts	Do this instead
Summary for latency	Cannot aggregate across replicas; no quantile flexibility	Histogram + histogram_quantile()
userID as a Prometheus label	Unbounded cardinality → Prometheus OOM	Hash to a bucketed user_tier or drop
Logging the same error you return	Duplicate lines, no single source of truth	Return wrapped, log once at the boundary
No RecordError on failed spans	Trace shows green, alert fires red	Always pair error returns with span.RecordError + SetStatus
Trace without context propagation	Each layer starts a new root trace	First argument of every I/O method is ctx context.Context
Global metric defined inside a handler	Re-registers on every call → panic	Declare metrics at package level, register once
Histogram with 200 buckets	High storage cost, slow queries	Start with DefBuckets, tune from real data

Verification Checklist

Before marking the change done:

• Each new code path emits at least one counter (operations) and one histogram (latency)
• All metric labels are bounded (method, route pattern, status class — not IDs)
• Each PromQL query the metric is meant to answer appears as a comment above the metric declaration
• Every I/O method takes ctx first; downstream calls pass it through
• Every service method, DB call, and outbound HTTP call has a span
• Failure paths call span.RecordError(err) and span.SetStatus(codes.Error, ...)
• At least one histogram uses ObserveWithExemplar for trace correlation
• Logs and traces are correlated (see go-logging skill: otelslog bridge)

References

• references/metrics.md — Counter/Gauge/Histogram, naming, cardinality, PromQL examples
• references/tracing.md — TracerProvider, spans, attributes, otelhttp, sampling
• references/correlation.md — Exemplars, trace_id in logs, end-to-end workflow
• references/anti-patterns.md — Detailed walkthrough of each anti-pattern with code