Observability and diagnostics

Observability and Diagnostics

Purpose

When something fails at 3am, the only witnesses are the logs — and most systems log either nothing useful or everything uselessly. This skill is the instrumentation discipline: every log line answers what happened, to which entity, and what the operator should do; levels are honest; skips are surfaced, not silent; and the acceptance test is brutal and simple — could you debug this failure from the output alone?

Metadata

• Prerequisites: none (loadable standalone); architecture-analysis helps place instrumentation at the right boundaries.
• Related Skills: debugging-playbook (consumes the evidence this skill produces), performance-engineering (production measurements ride on this instrumentation), configuration-management (secrets must never reach logs), agent-engineering (agent-run reconstructability applies this skill), engineering-minimalism (proportionality — metrics only when someone will look).
• Owns: structured logging; log levels; actionable error messages; surfaced skips; health/readiness signals; run reconstructability; correlation identifiers.

When to Use / When NOT to Use

Use when:

• Writing or reviewing any logging, error message, metric, or health check.
• A failure occurred and the output could not explain it — instrument before it recurs.
• Designing what a pipeline, job, service, or agent records about its own execution.
• Deciding whether to add metrics/tracing to a system.

Do NOT use (load the sibling instead):

• A failure is happening NOW and needs diagnosis → debugging-playbook (come back here to close the evidence gaps it exposes).
• The question is how fast/expensive, with a target → performance-engineering.
• The output problem is secrets appearing in logs → configuration-management step 6 owns secrets hygiene (this skill enforces it at the log line).
• A single-run CLI whose full output the operator watches live → minimal instrumentation is correct; don't build dashboards for a script (engineering-minimalism).

Definitions & Mental Model

• Structured log: a log event with named fields (timestamp, level, event, entity id, context) rather than an interpolated sentence — greppable and machine-parseable.
• The three questions: every WARNING-or-above line must answer (1) what happened, (2) to which entity (id/key/path), (3) what should the operator do about it.
• Honest levels: ERROR = an operator should act; WARNING = degraded but self-handled, worth knowing; INFO = lifecycle landmarks; DEBUG = everything else. A system that ERRORs routinely has trained its operators to ignore ERROR.
• Surfaced skip: when a pipeline skips a unit of work, the skip appears in the output (count and reason), never only in a debug log — silent skips misrepresent coverage.
• Run reconstructability: the property that a completed (or failed) run's story — inputs, decisions, outputs, termination reason — can be retold from its records alone.
• Correlation identifier: one id stamped on every record a single request/job/run produces, so its records can be isolated from the noise of concurrent work.

Mental model: write every log line for a stranger reading it during an incident, without the code open. That stranger has three questions (above) and no patience. Everything in this skill falls out of serving them: structure (so they can filter), honesty (so ERROR means something), entity ids (so they know which order/user/file), remedies (so they can act), and correlation (so they can follow one thread through the noise). And because that stranger is increasingly an AI agent reading logs programmatically, structure and explicitness pay double.

Core Methodology

1. Instrument the failure paths first. The happy path needs a start/finish landmark; the failure paths need the full three-questions treatment. Priority order mirrors risk: external calls (network, DB, subprocess, LLM), parsing of untrusted input, resource exhaustion points, and anything irreversible. If you instrument only one thing, instrument what you'll be asked about when it breaks.
2. Make every WARNING+ line answer the three questions.
   • Bad: ERROR: failed to process.
   • Good: ERROR event=invoice_send_failed invoice_id=INV-4411 reason=smtp_timeout remedy="check SMTP host config; retry with resend command". The remedy clause is the difference between a log and an alarm bell with a manual attached. For user-facing errors, the same craft applies to the message itself ("Run init first" — error-contract design is api-and-interface-design step 4; this skill owns the operator-facing wording).
3. Structure over prose. Named fields (key=value or JSON) for anything you will ever filter by: event name, entity id, duration, count, reason. Prose is for humans skimming; fields are for the grep/query that finds the needle. Use the logging facility the codebase already has (stdlib logging, a console library, print-to-stderr for a small CLI) — the discipline is the structure, not the framework (engineering-minimalism: don't add a logging stack to a 300-line tool).
4. Use levels honestly (decision tree):

Something notable happened. What level?
├─ An operator must act (data at risk, feature down, config broken)
│    → ERROR. And the line carries the remedy.
├─ The system handled it, but coverage/quality degraded
│  (item skipped, fallback used, retry succeeded)
│    → WARNING. Include what was skipped/degraded and the count.
├─ A lifecycle landmark (started, finished, N items in, M out, took T)
│    → INFO. Start and end of every job, with totals.
└─ Only useful when actively debugging → DEBUG. Default-off in prod.
Rule: if ERROR fires on every run, your levels are lying — demote
until ERROR means "act now".

5. Surface skips and degradations in the OUTPUT, not just the log. A job that processed 9,800 of 10,000 items must say so where its consumer looks: done: 9800 processed, 200 skipped (150 unparseable, 50 timeout). Per-item reasons go to WARNING; the aggregate goes in the result. Silent partial success is a correctness bug wearing a success exit code (the fatal-vs-recoverable framing is architecture-analysis's; this skill makes the recoverable path visible).
6. Stamp a correlation id through every multi-step flow. One id per request/job/run, present on every record it produces (and passed to sub-calls). Without it, concurrent runs interleave into an undebuggable braid. For agents: iteration number + action + truncated observation per cycle, plus the termination reason — the reconstructability requirement agent-engineering step 6 imposes is implemented with this skill.
7. Give long-running things a health signal. A liveness/readiness answer for services; a success-marker or heartbeat for scheduled jobs (so absence is detectable — the alert on a MISSING marker beats parsing logs for failure). Proportionality applies: a cron job needs a marker file and a second cron that checks it, not a monitoring platform.
8. Add metrics only when someone will look at them. A metric nobody reviews is instrumentation debt. The gate: name the person/agent and the decision the metric feeds. If the answer is "might be useful later," log the event instead — logs can be aggregated retroactively; unused dashboards cannot be un-built (engineering-minimalism rung 1).
9. Never log secrets, and bound log volume. Tokens, passwords, full payloads with PII: redact at the log call (secrets hygiene owned by configuration-management). Truncate large payloads (log the first N chars + length); unbounded log lines are their own outage. Per-item logs inside hot loops get sampled or aggregated.
10. Run the acceptance test. Take a recent failure (or inject one): can a person with only the logs/output — no code, no rerun — say what happened, to what, and what to do? If not, the gap they hit is the next instrumentation task. This test, applied quarterly or after every incident, is the whole skill in one question.

Instrumentation checklist

• Failure paths instrumented before happy paths; external calls all covered
• Every WARNING+ line: what happened + entity id + remedy
• Filterable facts are named fields, not prose
• ERROR is rare and actionable; job start/end at INFO with totals
• Skips/degradations aggregated into the visible output, with reasons
• One correlation id per run, on every record, passed to sub-calls
• Long-running work has a health signal whose ABSENCE is detectable
• Every metric names its consumer and the decision it feeds
• Secrets redacted at the call site; payloads truncated; hot loops sampled
• Acceptance test passed on a real or injected failure

Discovery & Audit Commands

Audit an existing codebase's diagnosability:

# What logging exists, and is it structured?
grep -rn -E "logger\.|logging\.|console\.(log|error|warn)|print\(" --include="*.py" --include="*.ts" --include="*.js" . | grep -v node_modules | head -30

# Honest levels? (ERROR that isn't actionable is the finding)
grep -rn -E "log.*error|logger\.error|console\.error" --include="*.py" --include="*.ts" . | grep -v node_modules | head -20

# Silent excepts / swallowed failures (the anti-observability smell)
grep -rn -E "except.*:\s*pass|catch\s*\(.*\)\s*\{\s*\}" --include="*.py" --include="*.ts" . | grep -v node_modules

# Are skips surfaced? (look for skip/continue paths without logging nearby)
grep -rn -B2 -A2 "continue" --include="*.py" . | grep -v node_modules | head -30

# Secrets near log calls (each hit needs eyeballing; -l to avoid echoing values)
grep -rliE "log.*(token|secret|password|key)" --include="*.py" --include="*.ts" . | grep -v node_modules

# Correlation ids in multi-step flows
grep -rn -iE "request_id|run_id|correlation|trace_id|session_id" --include="*.py" --include="*.ts" . | grep -v node_modules | head

# Health signals
grep -rn -iE "health|readiness|liveness|heartbeat" --include="*.py" --include="*.ts" . | grep -v node_modules | head

Failure Modes & Anti-patterns

Symptom	Mistake	Correction
Incident, and the logs say nothing useful	Happy path instrumented, failure paths bare	Failure-paths-first priority (step 1)
ERROR: something went wrong	The three questions unanswered	what + entity id + remedy on every WARNING+ (step 2)
Operators ignore ERROR (it always fires)	Dishonest levels	ERROR = act now; demote the rest (step 4)
Job "succeeded", 200 items silently missing	Skips logged at DEBUG or not at all	Aggregate skip counts + reasons in the visible output (step 5)
Two concurrent runs, interleaved unreadable logs	No correlation id	One id per run, on every record (step 6)
Nightly job silently stopped running weeks ago	Failure detectable only by reading logs that no longer appear	Success marker + absence alert (step 7)
Dashboard of metrics nobody has opened	Metrics without a consumer	Name the reader and the decision, or log instead (step 8)
Token printed in a stack trace	No redaction at the log boundary	Redact at call site; secrets hygiene per configuration-management (step 9)
40GB of logs, needle unfindable	Unbounded prose logging in hot loops	Structure + truncation + sampling (steps 3, 9)
except: pass around the flaky call	Failure swallowed to "keep it quiet"	Convert to WARNING with reason, or let it raise — never silence (steps 2, 5)
"Why did the agent do that?" unanswerable	Actions/observations not recorded	Per-cycle action+observation+termination logging (step 6)

Worked Example

Task: a document-import pipeline "works," but support keeps asking why particular documents are missing, and nobody can answer.

1. Acceptance test first: pick one missing document; try to explain it from current output. Current output: Import complete. Fail — total evidence gap.
2. Failure paths: three places drop documents: unparseable format, over-size limit, extraction timeout. Each currently a bare continue.
3. Instrument: each drop becomes WARNING event=doc_skipped doc_id=... reason=unparseable|oversize|timeout; the summary becomes INFO event=import_done run_id=... imported=4812 skipped=63 (41 unparseable, 19 oversize, 3 timeout) duration_s=212 — and the CLI prints the same aggregate line, so the skips are in the output, not just the log.
4. Correlation: each run gets a run_id; every line carries it — concurrent imports no longer interleave into confusion.
5. Health: the nightly run writes a dated success marker; a trivial second job alerts if the marker is absent by 06:00.
6. Levels: ERROR reserved for "storage unreachable — check DB credentials, imports halted" (operator must act); everything self-handled is WARNING.
7. Re-test: support asks about document 8841; grep doc_id=8841 → reason=oversize. Answered from logs alone, in one command. The pipeline's code didn't change behavior at all — it just stopped keeping secrets.

Repository Examples

Repo facts below are point-in-time illustrations (as of 2026-07-04) — examples, never assumptions about your system.

• ragit (~/ragit) — surfaced skips and actionable errors in a small CLI: unparseable files are skipped WITH an error message while indexing continues (the recoverable path made visible), and custom exceptions carry remedies — "Run ragit index {path} first". Also proportionality: a single-operator CLI uses console output, not a logging framework — correct for its size.
• agentix (~/agentix) — evidence preservation in an agent loop: tool failures return as observation strings ("Error running {action}: {e}") so the failure is in the transcript, not swallowed; the 2-strike malformed-JSON rule surfaces the RAW model output — the diagnostic evidence survives; explicit timeouts (shell 30s / python 15s / web 15s) make hangs bounded and attributable.
• prism (~/prism) — distinguishable failure classes and honest degradation: OllamaUnavailableError (act: is the backend running?) is separated from per-chunk timeouts (self-handled, chunk skipped); a missing GITHUB_TOKEN produces an explicit warning that reviews will be computed but not posted — degraded-and-said-so, never silent.

Validation Exercise (any repository): inject one failure into the system's most important external call (wrong port, bad credential); then, using only the produced output/logs, write three sentences: what happened, to which entity, what the operator should do. Every sentence you cannot write is an instrumentation task.

Validation Criteria

You applied this skill correctly when:

1. The acceptance test passes: a chosen failure is explainable from output alone, without the code.
2. Grepping the logs for any ERROR line yields text containing an entity id and a remedy.
3. A partial-success run's output states processed/skipped counts with reasons — verified by forcing a skip.
4. Two concurrent runs' records are separable by a single grep on the correlation id.
5. Killing the scheduled job produces a detectable absence (marker/heartbeat alert), not just silence.
6. A secrets grep over log call sites returns nothing unredacted, and no except: pass survives.

Provenance & Maintenance

• Sources: ~/ragit, ~/agentix, ~/prism — investigated 2026-07-04. The source corpus contains no metrics/tracing systems (single-operator local tools — proportionality applied), so metric/health guidance is general practice; the logging/skip/remedy guidance is repo-evidenced. Skill authored 2026-07-06.
• Assumptions: the three-questions rule and level semantics are conventions, not standards — teams with an established logging schema map onto it rather than replace it.
• Re-verification commands:

  grep -rn "skip\|unparseable" ~/ragit/ragit | head
  grep -rn "Error running" ~/agentix/agentix | head
  grep -rn "OllamaUnavailable\|warning" ~/prism/prism | head
  

• Likely to drift: the example repos' logging idioms; observability tooling landscape (OpenTelemetry et al.) if the owner's systems grow beyond single-operator scale.
• Maintenance checklist:
  • Re-run re-verification; re-stamp Repository Examples.
  • If any owner system gains metrics/tracing, add it as the missing case study.
  • Confirm cross-referenced skills still exist under their directory names.