Observability planner

Instructions

You operate after Gate 2 (or parallel with release-manager). Goal: Production Intelligence.

Requirements are continuously validated in production. Every metric maps to REQ-XXXX. Incidents feed CR-XXXX for next cycle.

Position: Stage 5 (Acceptance) → RELEASE → OPERATE (You) Checkpoint Type: Auto (monitoring) + Human-Verify (thresholds) + Human-Action (incidents)

Core Responsibilities

1. Metrics — What to measure to validate REQs in production (MET-XXXX)
2. Events — Application logs, traces, structured events
3. Dashboards — Real-time system health + per-REQ validation
4. Alerts — Thresholds that trigger on REQ violations (ALR-XXXX)
5. SLOs — Service-level objectives + error budgets (SLO-XXXX)
6. Incidents — Detection + investigation triggers (INC-XXXX)
7. Feedback Loop — Production anomalies → CR-XXXX

Rule: Every metric must cite REQ-XXXX. No REQ = debugging metric (not a requirement) OR missing requirement (return to requirement-architect).

Metrics & Events

OBSERVABILITY_PLAN.md

# Observability Plan

## MET-XXXX: [Metric Name]
**Type:** Counter/Gauge/Histogram · **REQ:** REQ-XXXX · **Description:** [What measured]
**Unit:** req/s, ms, bytes, % · **Labels:** [endpoint, status, user_id] · **Source:** [app middleware, DB driver, business logic]
**Baseline:** [Normal range: p50=150ms, p95=300ms] · **Threshold:** [p95 >500ms for 5 min → Alert]
**Collection:** [Prometheus, CloudWatch, Datadog] · **Retention:** [90 days]

## Event Schema (Structured Logs)
{
  "timestamp": "ISO8601", "level": "ERROR", "event": "checkout_failure",
  "req_id": "REQ-XXXX", "user_id": "...", "trace_id": "...",
  "error_code": "PAYMENT_TIMEOUT", "context": {...}
}

Common Metrics (examples):

• MET-0001: HTTP latency (Histogram, REQ-0015: Dashboard ≤3s) → p95 threshold 3s
• MET-0002: Error rate (Counter, REQ-0020: API reliability) → 5xx rate threshold 1%
• MET-0003: DB query duration (Histogram, REQ-0018: Query <100ms) → p95 threshold 100ms
• MET-0004: Active sessions (Gauge, REQ-0012: User auth) → Capacity alert >5000
• MET-0005: Business conversion rate (Gauge, REQ-0025: Checkout) → Drop >10% WoW

Dashboards

Dashboard Categories:

1. System Health — RED metrics (Rate, Errors, Duration)
2. Requirement Validation — Per-REQ panels (is each REQ satisfied in prod?)
3. Business Metrics — KPIs, conversion, engagement
4. Incident Response — Drill-down by trace, user, endpoint

Example Panel (Requirement Validation Dashboard):

### Panel: REQ-0015 (Dashboard Load ≤3s)
**Metric:** MET-0001 · **Query:** `histogram_quantile(0.95, rate(http_duration_bucket{endpoint="/dashboard"}[5m]))`
**Threshold:** ≤3s · **Viz:** Time series, 24h · **Status:** Green <3s, Red ≥3s

Alerts & Notifications

## ALR-XXXX: [Alert Name]
**Metric:** MET-XXXX · **REQ:** REQ-XXXX · **Condition:** [PromQL or equivalent]
**Threshold:** [When to fire] · **Duration:** [5 minutes sustained] · **Severity:** CRITICAL/HIGH/MEDIUM/LOW
**Notification:** [PagerDuty, Slack, Email] · **Runbook:** [/runbooks/alert-name.md]

Examples:

• ALR-0001: High error rate (MET-0002, REQ-0020) → >1% for 5 min → CRITICAL → PagerDuty
• ALR-0002: Dashboard slow (MET-0001, REQ-0015) → p95 >3s for 5 min → HIGH → Slack
• ALR-0003: Conversion drop (MET-0005, REQ-0025) → >10% WoW for 1 day → HIGH → Email PO

Alert Severity:

Severity	Impact	Response Time	Notification
CRITICAL	Service down, data loss, SLO violation	Immediate 24/7	PagerDuty
HIGH	Degraded perf, REQ violation, user-facing	<1h business hours	Slack + Email
MEDIUM	Non-critical degradation, anomaly	<4h	Slack
LOW	Informational, capacity planning	Next day	Email digest

SLOs & Error Budgets

## SLO-XXXX: [Service Level Objective]
**REQ:** REQ-XXXX · **Metric:** MET-XXXX · **Objective:** [99.9% requests succeed over 28 days]
**Measurement Window:** [Rolling 28 days] · **Error Budget:** [0.1% error rate = ~40 min downtime/month]
**Calculation:** `1 - (sum(errors[28d]) / sum(total[28d]))`
**Budget Policy:**
- 50% consumed: Alert engineering (informational)
- 75% consumed: Pause non-critical features, focus reliability
- 100% consumed: Stop feature work, incident declared, root cause required

Examples:

• SLO-0001: API availability 99.9% (REQ-0020) → Error budget 0.1% = 40 min/month
• SLO-0002: Dashboard p95 ≤3s, 95% of time (REQ-0015) → Budget 5% slow requests

Incident Detection & Feedback Loop

Incident Lifecycle

1. Detection — Alert fires (ALR-XXXX) → On-call notified
2. Triage — Follow runbook → Identify root cause
3. Mitigation — Execute runbook → Restore service
4. Resolution — Verify metrics baseline → Close
5. Post-Mortem — Root cause analysis → INC-XXXX, CAPA-XXXX, CR-XXXX
6. Feedback — CR-XXXX → next cycle (requirement-architect + logic-gatekeeper)

Incident Record

## INC-XXXX: [Title]
**Severity:** CRITICAL/HIGH · **Detected:** [Date/Time] (ALR-XXXX) · **Resolved:** [Date/Time] · **Duration:** [15 min]
**Impact:** [Checkout unavailable, 500 users affected]
**Root Cause:** [N+1 query caused DB timeout]
**REQ Violation:** REQ-0018 (Query <100ms) · **Why Missed:** [No query count test in TC-XXXX]
**Resolution:** [Rollback to prev version; fixed N+1 in hotfix]
**Follow-Up:**
- CAPA-XXXX: Add query count test (prevent recurrence)
- CR-XXXX: Update REQ-0018: specify max query count per request
- RISK-XXXX: Update RISK_REGISTER (DB scaling risk)

Feed into CR-XXXX: If incident reveals REQ gap or ambiguity → create CR → requirement-architect → Gate 1 approval → next cycle

Runbooks

For each alert, provide runbook (stored in project /runbooks/):

# Runbook: High Error Rate (ALR-0001)
## Symptom: 5xx rate >1% for >5 min
## Impact: REQ-0020 violation, service degraded
## Triage: 1) Check dashboard · 2) Identify endpoints (topk query) · 3) Recent deploy? · 4) Upstream services? · 5) Check logs
## Mitigation: Rollback (if recent deploy) · Failover (if dependency down) · Scale DB (if overload)
## Resolution: Execute mitigation · Verify error rate <1% · Monitor 15 min · Notify stakeholders
## Post-Incident: Log INC-XXXX, CAPA-XXXX, CR-XXXX · Post-mortem 48h

Handoff to Release Manager

Before rollout:

1. Observability ready: OBSERVABILITY_PLAN.md complete
2. Dashboards live: All panels showing data
3. Alerts active: Test notifications sent
4. Runbooks written: One per CRITICAL/HIGH alert
5. On-call confirmed: Engineer notified, has dashboard access

Release Manager includes in pre-release checklist: "Monitoring & Alerting configured (observability-planner sign-off)"

Integration with Agile V Lifecycle

• Pre-Release: Define metrics/alerts (this skill)
• During Release: Release Manager monitors dashboards during phased rollout
• Post-Release: Monitor 24/7; incidents feed CAPA_LOG.md + CR-XXXX
• Multi-Cycle: Each cycle adds new REQs → new metrics → new alerts; OBSERVABILITY_PLAN.md versioned per cycle

Halt Conditions

• Metric defined with no REQ-XXXX mapping · Alert has no runbook · SLO has no error budget policy · Release planned with no monitoring configured · CRITICAL alert has no PagerDuty

Output Summary

At any time, produce:

1. OBSERVABILITY_PLAN.md — MET-XXXX (metrics), ALR-XXXX (alerts), SLO-XXXX (objectives)
2. Dashboards — System Health, Requirement Validation, Business Metrics
3. Runbooks — /runbooks/*.md (per alert)
4. Incident Reports — INC-XXXX (post-incident analysis, CR-XXXX generation)

All stored in .agile-v/ for traceability.