Observability debugging

Observability Debugging

Use this skill when the fastest path to understanding a problem is through runtime signals rather than code inspection alone.

Goal

Increase debugging success rate by using observability signals to narrow uncertainty.

Focus on:

1. what signals are available
2. which signals are noise vs useful indicators
3. how different signals correlate in time and scope
4. what likely path or subsystem they point to
5. what is still unobservable or missing
6. what the next debugging step should be

Core Rules

1. Start from signals, not assumptions

When logs, metrics, traces, or telemetry exist, begin with the actual observed signals:

• log lines
• error patterns
• request/response timing
• counters, rates, spikes, drops
• trace spans or step durations
• browser-visible timing and runtime output

Do not start with a code theory before checking what the system already reveals.

2. Separate signal from symptom

A useful signal is not always the same as the user-visible symptom.

Distinguish between:

• symptom: what the user or operator noticed
• signal: what the system emitted
• indicator: the signal most useful for narrowing the problem
• gap: what you still cannot see because instrumentation is missing or weak

3. Correlate before concluding

One log line or one spike is often not enough.

Look for relationships across:

• time
• route/feature/workflow
• request or task boundaries
• error bursts vs latency changes
• release/deploy timing
• user-visible failures vs backend/runtime signals

4. Prefer narrowing over exhaustive reading

Do not read every log or inspect every metric blindly.

Narrow by:

• time window
• feature or route
• failing command or request type
• severity or error class
• before/after release window
• affected environment only

5. Use observability to choose the next path

The point is not just to collect signals, but to make a better debugging decision.

Good next paths include:

• deeper root-cause tracing in one subsystem
• deployment verification for environment mismatch
• performance debugging for slow paths
• incident triage if impact is broader than expected
• adding instrumentation where visibility is too weak

6. Notice missing instrumentation explicitly

If the available signals are not enough, say so.

Call out gaps such as:

• no clear error classification
• no timing around the suspected step
• no request correlation
• no release marker around the failure window
• logs too noisy to isolate the failing path

Missing visibility is itself a useful finding.

Minis-Specific Workflow

Shell-first evidence gathering

Use shell commands to:

• inspect logs and outputs
• search for repeating error patterns
• narrow time windows or message classes
• compare before/after output snapshots
• inspect request, job, or command-level traces when available

Browser-visible observability

For web apps:

• compare user-visible timing with underlying logs or outputs
• separate rendering lag from backend or network delay
• connect visible broken states with emitted runtime signals where possible

Compact-signal bias

On Minis, prefer compact, decision-oriented observability review.

A good first pass should answer:

• what changed
• what spiked or degraded
• what path seems most implicated
• what is still unknown

Investigation Sequence

Use this order:

1. Restate the symptom.
2. Identify available signals.
3. Narrow the time window, path, or failing workflow.
4. Find the most useful indicators.
5. Correlate indicators across sources.
6. State what the signals most strongly suggest.
7. State what remains unobservable.
8. Choose the next debugging or mitigation path.

Common Observability Mistakes

Watch for:

• staring at raw logs without narrowing scope
• treating one log line as a full explanation
• confusing high volume with high importance
• ignoring timing relationships around releases or user-visible failures
• assuming missing errors means no problem exists
• not stating when instrumentation is insufficient

Integration with Other Skills

• Use incident-triage if the signals suggest broader or actively growing impact.
• Use root-cause-tracing after observability narrows the likely subsystem or path.
• Use performance-debugging when the dominant signals are latency or resource related.
• Use deployment-verification if signals suggest environment-specific behavior.
• Use postmortem-writing later to capture observability lessons and instrumentation gaps.

Completion Checklist

Before calling the observability review done, check:

• Is the symptom clearly stated?
• Were the available signals listed?
• Was the scope narrowed sufficiently?
• Were the most useful indicators identified?
• Were correlations described, not just raw data copied?
• Were instrumentation gaps called out?
• Is the next debugging path explicit?

Response Template

• Symptom:
• Signals reviewed:
• Narrowed scope:
• Most useful indicators:
• Correlations observed:
• Likely implication:
• Missing visibility:
• Recommended next path:

Example Triggers

• "Use the logs and runtime signals to narrow this issue."
• "What do these metrics and errors actually suggest?"
• "Correlate these signals and tell me where to debug next."
• "The code path is unclear — use observability first."
• "Help me distinguish noisy telemetry from useful evidence."