Observability and alerting

Observability and Alerting

Overview

Observability is designed in, not bolted on after the first outage. The goal isn't dashboards — it's the ability to answer "is the system OK, and if not, where and why" in minutes, at 3 AM, by whoever is on call.

Three pillars, three jobs: metrics detect (cheap aggregations over time), logs explain (detailed events for investigation), traces follow (one request's path across services). You need all three, and you need to know which question each answers.

Alert on symptoms, not causes. "Users can't check out" pages someone. "CPU at 81%" usually doesn't — it's a cause to investigate after a symptom fires. Cause-based alerting is how you get 400 ignored alerts and one missed outage.

When to Use

• Instrumenting a new service (metrics, logs, traces)
• Setting up Prometheus, Grafana, Loki, Tempo/Jaeger, OpenTelemetry
• Writing or reviewing alert rules and Alertmanager routes
• Defining SLIs, SLOs, and error budgets
• Reducing alert fatigue
• Pre-launch observability review

The Four Golden Signals (measure these first)

For every user-facing service:

1. Latency — how long requests take (distinguish successful vs failed request latency)
2. Traffic — demand on the system (requests/second, messages/second)
3. Errors — rate of failed requests (explicit failures + wrong-content successes)
4. Saturation — how "full" the system is (CPU, memory, connection pools, queue depth)

These four detect nearly every user-impacting problem. Everything else is supporting detail. (For infrastructure resources, USE: Utilization, Saturation, Errors. For request-driven services, RED: Rate, Errors, Duration.)

The Reference Stack

Applications ──(OTel SDK /agent)──▶ Grafana Alloy / Fluent Bit / Vector (collection)
                                        │
                    ┌───────────────────┼────────────────────┐
                    ▼                   ▼                    ▼
              Prometheus            Loki               Tempo/Jaeger
              (metrics)             (logs)             (traces)
                    └───────────────────┼────────────────────┘
                                        ▼
                                    Grafana          ← dashboards, explore, correlation
                                        ▼
                                  Alertmanager       ← routing, silencing, grouping

• Prometheus — metrics scraping and storage; recording rules for expensive queries
• Grafana — visualization and the single pane for correlating all three signals
• Loki — log aggregation (label-indexed, not full-text — cheap and pairs with Prometheus labels)
• Tempo/Jaeger — distributed tracing
• OpenTelemetry — the instrumentation standard: one SDK/agent, any backend. Never instrument directly against a vendor API.
• Alertmanager — dedup, grouping, routing, silences
• Uptime Kuma (or blackbox exporter) — simple external availability checks; outside-in monitoring that doesn't depend on the system it monitors

Instrumentation Standards

Metrics:

• Expose /metrics (Prometheus format) from every service; standard labels: service, env, version, instance.
• Histograms for latency (need percentiles — averages lie); counters for requests and errors.
• Cardinality discipline: never label by user ID, request ID, or email. Unbounded cardinality kills Prometheus. IDs belong in logs and traces, not metric labels.

Logs:

• Structured (JSON) always: {"ts":"...","level":"info","service":"api","msg":"order created","order_id":"..."} — greppable and parseable.
• Levels used honestly: ERROR means a human should probably know; DEBUG off in prod by default, enable per-service when hunting.
• Never log secrets, credentials, or PII without a documented legal basis and masking.
• Include the trace ID in every log line — that's the thread that ties all three signals together.

Traces:

• OpenTelemetry auto-instrumentation first; manual spans around the operations that matter (DB calls, external APIs, queue hops).
• Sample deliberately: 100% of errors, a percentage of successes, all of a canary's traffic.

SLIs, SLOs, and Error Budgets

SLI  = the measurement from the user's perspective
       e.g., "fraction of checkout requests completing successfully in < 500ms"
SLO  = the target over a window
       e.g., "99.9% over 30 days"
Error budget = 1 − SLO = 0.1% of requests may fail
             ≈ 43 minutes of full downtime per month

Why it matters: the error budget converts reliability from a vibe into a decision-making tool. Budget remaining → ship faster. Budget burned → freeze risky changes and invest in reliability. It ends the "dev speed vs ops stability" argument with arithmetic.

Rules:

• SLOs are set on user experience, not infrastructure ("checkout succeeds," not "pods are up").
• Start with a few SLOs on the critical user journeys, not 50 on everything.
• SLOs are agreed with product/business owners — they define what "enough reliability" costs.

Alert Design

Every alert must pass this test:

# Good alert: symptom, impact, actionable
- alert: CheckoutErrorBudgetBurn
  expr: |
    (
      sum(rate(checkout_requests_total{status=~"5.."}[1h]))
      /
      sum(rate(checkout_requests_total[1h]))
    ) > 14.4 * 0.001   # fast burn: 2% of monthly budget in 1 hour
  for: 5m
  labels:
    severity: page
    team: shop
  annotations:
    summary: "Checkout failing for users"
    impact: "Users cannot complete purchases"
    runbook: "https://wiki/runbooks/checkout-errors"
    dashboard: "https://grafana/d/checkout"

The mandatory alert contents:

1. What's happening (symptom, not "CPU high")
2. User impact (so the responder can prioritize at 3 AM)
3. Runbook link (what to do — an alert without a runbook is just anxiety)
4. Dashboard link (where to look)

Severity tiers:

• page — user impact NOW; wake someone up
• ticket — needs attention in business hours (error budget burn at slow rate, disk filling in days)
• record — interesting, no notification (feeds dashboards and postmortems)

Alertmanager hygiene: group related alerts, route by team label, silence with expiry dates and reasons, and dead-man switches for the monitoring system itself (who watches the watchers: external uptime checks).

Alert Fatigue: The Silent Killer

Rules that keep alerting honest:

1. Every page requires human action. If the correct response is "wait and see," it's a ticket, not a page.
2. Alerts that fire and auto-resolve with no action get fixed or deleted. Review alert history monthly: pages per week, acknowledged-but-no-action rate.
3. Flapping alerts (firing > 3 times/day) are P1 tech debt — they train responders to ignore the pager.
4. Target: < 2 actionable pages per week for a healthy service. More means either the service or the alerting is broken.
5. New alerts start as ticket severity and earn page status.

Retention With Intent

Data	Typical retention	Why
Metrics (high-res)	15–30 days	Recent debugging
Metrics (downsampled)	13 months	Trends, capacity planning
Operational logs	30–90 days	Debugging windows
Audit/security logs	1+ year (per compliance)	Forensics, regulation
Traces	7–30 days	Recent request forensics

Indefinite retention of everything is a cost problem and a liability problem (PII). Retention is a decision, recorded, not an accident of defaults.

Pre-Launch Observability Gate

Nothing launches until all of these are true:

• Four golden signals dashboarded per service
• At least one SLI/SLO on the critical user journey
• Alerts exist for the top 3 failure modes, each with a runbook link
• Structured logs flowing with trace IDs
• Traces across service boundaries
• External uptime check (someone outside the cluster watches the front door)
• On-call routing reaches a human, tested with a real test alert
• Dashboards verified during a staging failure drill — can someone unfamiliar find the problem in < 5 minutes?

Common Rationalizations

Rationalization	Reality
"We'll add monitoring after launch"	The first outage arrives before the "after launch" task does. Observability is a launch requirement, not a follow-up.
"Alert on everything, just in case"	200 ignored alerts = 0 alerts. Every non-actionable alert dilutes the ones that matter.
"CPU alerts have served us fine"	CPU at 90% with happy users is fine; CPU at 20% with users getting 500s is an outage. Symptoms page; causes inform.
"We don't need SLOs, we just need it to work"	Without an SLO, "works" is undefined, and every reliability investment is an argument instead of a budget decision.
"Averages are enough for latency"	Average latency hides the p99 that your biggest customers hit. Histograms or nothing.
"Let's keep all logs forever, storage is cheap"	It isn't, and unbounded PII retention is a compliance incident. Retain with intent.

Red Flags

• Alerts with no runbook links, or runbooks that say "call Dave"
• Pager history showing > 5 pages/week per service, most auto-resolving
• Dashboards nobody has opened in months
• No traces; debugging microservices by grep-ing 12 log streams by timestamp
• Metrics with user IDs or emails as label values
• "Monitoring" = a ping check on the homepage
• The monitoring stack monitors itself and nothing else (or vice versa: no dead-man switch)
• SLOs don't exist, so every incident becomes a debate about whether it's "bad enough" to invest in reliability

Verification

• Four golden signals visible per service in Grafana
• SLIs/SLOs defined for critical journeys; error budget visible on a dashboard
• Every alert has: symptom, impact, runbook link, dashboard link, correct severity
• Logs structured JSON with trace IDs; secrets/PII absent
• Traces span service boundaries; exemplars link metrics → traces
• External availability monitoring independent of the platform
• Monthly alert review evidence: pages/week, no-action rate, flapping fixes
• A new team member successfully diagnosed a staging failure using only the dashboards and runbooks