Observability

Observability

Design and audit the signals a running system emits so failures are visible before users report them. This skill builds the standing pipeline - instrumentation, metrics, traces, structured logs, alert rules, SLOs, and dashboards-as-code - and audits a repo for the gaps that leave a service blind. It produces config (exporters, recording/alerting rules, OTLP pipelines, dashboard JSON), so the AI Self-Check applies.

Target versions (July 2026):

• Prometheus: 3.13.1
• OpenTelemetry Collector: v0.156.0
• Grafana: 13.1.1
• Grafana Loki: 3.7.3
• Grafana Tempo: 3.0.2
• Alertmanager: 0.33.1

When to use

• Adding instrumentation to a service: metrics (Prometheus/OTLP), traces (OpenTelemetry), or structured logs
• Writing or reviewing alert rules, recording rules, and Alertmanager routing
• Defining SLOs and error budgets, and the multi-burn-rate alerts that back them
• Building dashboards as code (provisioned JSON, grafonnet/Grizzly)
• Designing the signal collection layer: OTel Collector pipelines, exporters, scrape config
• Auditing a repo for observability gaps: uninstrumented services, no SLOs, alert-fatigue patterns, cardinality risks, logs with no trace correlation

When NOT to use

• Checking whether a live cluster is healthy right now (point-in-time, read-only diagnostics) - use cluster-health
• Writing or reviewing Kubernetes manifests, Helm charts, or the Prometheus Operator CRDs as K8s objects - use kubernetes
• Wiring CI/CD pipelines (the skill defines what they should emit and gate on, not the pipeline itself) - use ci-cd
• Localizing an unknown-layer live failure once signals exist (consuming signals to find root cause) - use debug-triage
• Application security review or secret scanning in telemetry - use security-audit

---

AI Self-Check

AI tools produce the same observability mistakes. Before returning any generated instrumentation, rule, pipeline, or dashboard, verify:

• Metric cardinality bounded: no unbounded label values (user IDs, request paths with IDs, timestamps, full URLs) on metrics. High cardinality is the top cause of Prometheus OOM.
• Rules validated: PromQL/alert rules pass promtool check rules; routing passes amtool config routes. AI invents plausible-but-wrong PromQL functions and label matchers.
• Alerts are actionable: every alert has for:, severity, a runbook link, and fires on symptoms (SLO burn, user-facing error) not raw causes. No alert that a human cannot act on.
• SLO math is real: error budget = 1 - SLO; burn-rate alerts use multi-window multi-burn-rate, not a single threshold. State the window and budget explicitly.
• Trace context propagated: W3C traceparent propagation is configured end to end; spans carry service.name. Logs include trace_id/span_id for correlation.
• No secrets in telemetry: no tokens, auth headers, PII, or full request bodies in span attributes, log fields, or metric labels.
• Versions and signals real: exporter names, OTLP receiver/exporter names, PromQL functions, and Grafana panel types verified against current docs - not assumed.
• Sampling intentional: trace sampling rate is stated and justified (head vs tail), not silently defaulted; 100% sampling on a hot path is flagged.

---

Workflow

Step 1: Identify the signals and the questions

Pin down what the system must answer before choosing tools. For each service: what does "broken" look like to a user, and which signal proves it? Map to the golden signals (latency, traffic, errors, saturation) or RED (rate, errors, duration) for request-driven services and USE (utilization, saturation, errors) for resources. Pick the minimum signal set that answers those questions - do not instrument everything because you can.

Step 2: Choose the collection path

Need	Default
Metrics	Prometheus scrape, or OTLP metrics through the OTel Collector to a Prometheus-compatible store
Traces	OpenTelemetry SDK -> OTLP -> Collector -> Tempo (or vendor backend)
Logs	Structured JSON -> agent (Alloy/Promtail/OTel) -> Loki
Unified pipeline	OpenTelemetry Collector as the single ingest/route/transform layer

Prefer OTLP and the OTel Collector as the vendor-neutral seam: instrument once, re-route backends in config. Use direct Prometheus scrape where pull and existing exporters already fit.

Step 3: Instrument and configure

• Metrics: use auto-instrumentation where it exists; add custom metrics only for domain-specific questions. Keep labels low-cardinality. Add recording rules for expensive queries that dashboards or alerts repeat.
• Traces: enable context propagation, set service.name and resource attributes, choose a sampling strategy (head sampling at the SDK, or tail sampling in the Collector for error/latency-biased retention).
• Logs: emit structured JSON, include trace_id/span_id, avoid logging what a metric already counts.
• Alerts and SLOs: write symptom-based alert rules, define SLOs with explicit windows, back them with multi-window multi-burn-rate alerts, route by severity in Alertmanager.
• Dashboards: keep them as code (provisioned JSON or grafonnet) so they are reviewable and reproducible, not click-built.

Step 4: Validate

• promtool check config / promtool check rules for Prometheus config and rules
• promtool test rules for unit tests on alerting/recording rules against sample series
• otelcol validate --config for Collector pipelines
• amtool config routes test / amtool check-config for Alertmanager routing
• Confirm a test signal traverses the full path (emit -> collect -> store -> query -> alert) on at least one service before declaring coverage
• Start with references/runnable-examples.md when a task needs compact Collector, SLO-rule, rule-test, or dashboard artifacts that can be passed to the validators above.

---

Signals reference

Metrics

• Naming: unit-suffixed, base units (seconds, bytes), _total for counters. Follow Prometheus and OpenTelemetry semantic conventions; do not invent metric names where a convention exists.
• Cardinality is the budget. Series count = product of label-value sets. Keep label values bounded and finite. Per-entity detail (user, request, order, full path) belongs on a trace attribute or log field, never a metric label. Exemplars link a metric sample to a trace - enable them for latency histograms.
• Recording rules precompute heavy expressions; alerting rules fire on conditions. Keep them in version control and unit-test them with promtool test rules.

Traces

• One trace = one request across services, stitched by propagated context. Without propagation you get disconnected spans, not traces.
• Sampling: head sampling is cheap and simple but blind to rare errors; tail sampling (in the Collector) keeps error/slow traces at the cost of buffering. State which and why.
• TraceQL queries Tempo; exemplars and trace_id in logs are the cross-signal jumps that make a trace findable from a metric spike or a log line.

Logs

• Structured over free text: JSON fields are queryable (LogQL), prose is grep-only. Include service, level, trace_id, span_id, and a stable message key.
• Logs are the most expensive signal per byte of insight. If a metric can answer it, count it; reserve logs for the context a metric cannot carry.

Alerts and SLOs

• An SLO is a target on an SLI (e.g. 99.9% of requests < 300ms over 30 days). Error budget is the allowed failure: 1 - SLO. Alert on budget burn rate, not on every breach.
• The burn-rate alert threshold is burn_rate * (1 - SLO) compared against the error-ratio SLI - not the raw error rate against a bare multiplier (a common off-by-budget bug). Standard tiers for a 99.9% SLO: fast-burn 14.4x (1h + 5m windows, page), slow-burn 6x (6h + 30m, page), erosion 3x (24h + 2h, ticket). Both windows in a tier must breach together before the alert fires - express as an and: ratio_rate1h > 14.4*(1-SLO) and ratio_rate5m > 14.4*(1-SLO).
• Multi-window multi-burn-rate alerting (fast-burn + slow-burn windows) catches both acute outages and slow erosion while suppressing flapping. A single static threshold does neither.
• Alert hygiene: page only on user-impacting symptoms with a runbook; everything else is a ticket or a dashboard. Alert fatigue is an outage you stop seeing.

Dashboards as code

• Provision dashboards from version-controlled JSON or generate them with grafonnet/Grizzly. A click-built dashboard is an undiffable, unreviewable, un-restorable artifact.
• Committed-in-git is necessary but not sufficient: an exported JSON blob (regenerated on each export, never hand-edited) still drifts from the running dashboard and diffs unreadably. The source of truth is authored or generated config that flows file -> Grafana, not the reverse.

---

Audit lens (Wave 3 in deep-audit)

When auditing a repo for observability, report findings on:

• Coverage gaps: services that emit no metrics/traces/logs; endpoints with no latency or error signal; background jobs with no success/failure metric.
• No SLOs / no error budget: alerting exists but is threshold-based with no SLO backing.
• Alert anti-patterns: cause-based alerts with no for:, no runbook, no severity; duplicate or flapping alerts; paging on non-actionable conditions.
• Cardinality risk: unbounded labels (IDs, paths, emails) on metrics; high-cardinality log fields used as metric labels.
• Broken correlation: logs without trace_id; traces without service.name; metrics without exemplars on key histograms.
• Untested rules: alert/recording rules with no promtool test rules coverage.
• Drift risk: dashboards stored as exported blobs nobody edits, or not in version control.

Report only what the repo files show. Do not assume a running backend exists; flag "signal defined but no evidence it is collected" as a gap, not a pass.

---

Output Contract

See references/output-contract.md for the full contract.

• Skill name: OBSERVABILITY
• Deliverable bucket: audits
• Mode: conditional. When invoked to audit a repo for observability gaps (the Wave 3 lens), emit the full contract - boxed inline header, body summary inline plus per-finding detail in the deliverable file, boxed conclusion, conclusion table - and write the deliverable to docs/local/audits/observability/<YYYY-MM-DD>-<slug>.md. When invoked to build instrumentation, rules, pipelines, or dashboards, respond freely without the contract.
• Severity scale: P0 | P1 | P2 | P3 | info (see shared contract; only used in audit mode).

Related Skills

• cluster-health - point-in-time, read-only Kubernetes diagnostics ("is it healthy now"). Observability builds the standing signal pipeline ("can we see it over time"). cluster-health reads signals live; observability defines and audits them.
• kubernetes - authors manifests, Helm, and Operator CRDs as K8s objects. Observability authors the instrumentation, rules, and SLO/alert config those objects carry, platform-agnostic.
• debug-triage - consumes signals to localize an unknown-layer live failure. Observability produces the signals it consumes: producer vs consumer.
• ci-cd - wires the pipeline. Observability defines what the pipeline should emit and gate on, not the pipeline itself.
• security-audit - reviews exploitable vulnerabilities and secret exposure. Observability flags secrets-in-telemetry as a gap but does not replace a security review.

Rules

1. Cardinality is a hard budget. Never put unbounded values (IDs, paths, emails, timestamps) in metric labels. Bounded label sets only.
2. Validate before returning. Run promtool, otelcol validate, and amtool on generated config and rules; do not ship unverified PromQL or routing.
3. Alert on symptoms with runbooks. Every alert is actionable, has for: and severity, and links a runbook. No cause-only or non-actionable pages.
4. SLO-back the alerts. Define SLOs with explicit windows and use multi-window multi-burn-rate alerting, not single static thresholds.
5. Propagate context. Configure W3C trace propagation, service.name, and trace_id/span_id in logs so signals correlate.
6. No secrets in telemetry. No tokens, PII, auth headers, or full bodies in labels, span attributes, or log fields.
7. Dashboards as code. Provision from version control; never treat a click-built dashboard as the source of truth.
8. Run the AI Self-Check before returning any generated instrumentation, rule, pipeline, or dashboard.
9. Verify versions and signal names. Confirm exporter/receiver names, PromQL functions, and panel types against current docs; pin versions with dates.