Analytical troubleshooting

Analytical Troubleshooting

A staged method for finding the cause of a deviation: performance that used to be acceptable (or should be) no longer is, and nobody knows why.

The core discipline: track what the problem IS and, with equal care, what it plausibly could be but IS NOT. A cause that explains only the failures is a guess; a cause that explains the failures and the survivals is a diagnosis. Most troubleshooting failure — human and model alike — comes from anchoring on the first plausible cause and collecting only confirming evidence. This method makes that structurally hard to do.

Why structure instead of intuition: the evidence says procedural scaffolding (tables, gates, checklists) outperforms both raw expertise and good intentions. Domain-theory knowledge does not predict troubleshooting success; maintaining evidence discipline and switching strategies does. Your job is to be the process leader and bookkeeper. When a human partner is involved, they are the sensors and hands; deliberately keep the roles that way — a process-leading non-expert asking sharp questions is a proven pattern precisely because it resists the expert's urge to assume.

Hard rules (they exist because models measurably break them)

1. Never invent evidence. Every fact in the analysis carries a provenance tag: [observed] (directly seen this session by agent or user), [reported] (someone said so), or [assumed]. Evidence not obtained from the world must not be recorded as observed. When a test hasn't run yet, its result is unknown — not "probably fine".
2. No fix before a surviving cause. Do not propose remediation until at least one candidate cause has survived the paper test (Stage 4). Quick diagnostic actions are always fine; "let's just reinstall/replace/reboot and hope" is the failure mode this skill exists to prevent. (Exception: genuine safety/stop-the-bleeding containment — say explicitly that it is containment, not diagnosis.)
3. Every planned test names its targets. Before running or requesting any test, state which candidate causes it could refute. A test that can only confirm the favorite hypothesis is close to worthless; forcing the refutation question measurably improves diagnosis.
4. The spec table is the analysis. Maintain it continuously and re-emit the current table whenever it changes materially or the conversation grows long. Long sessions silently lose mid-context evidence; the re-emitted table is the antidote. For multi-session work, persist it to a file.
5. If nothing survives, tighten the spec — not the story. When every candidate fails testing, the specification is missing data. Go get more facts. Do not relax the standard until a pet theory passes.

Stage 0 — Triage

Before analyzing anything, make sure there is one analyzable problem.

• Separate. "The cluster is broken" is usually several deviations. Different symptoms, different objects, different timelines → separate analyses. Bundled deviations contaminate each other's evidence; splitting them is often the single move that unsticks a stalled investigation.
• Prioritize multiple deviations by current impact, urgency, and growth potential. Fix the bleeding first, analyze the important.
• Entry gate. Full analysis is warranted only if all three hold:
  1. There is a real deviation (a stateable SHOULD vs ACTUAL gap).
  2. The cause is unknown.
  3. Knowing the cause matters for acting well (if a rebuild is cheap and acceptable, do that — say so honestly).
• Route non-deviations elsewhere. Choosing between options is a decision, not a problem. Learning from a resolved incident is a retrospective (hand off to understanding-human-error). Process variation ("yield wobbles 2–4%") needs statistical methods. See references/edge-cases.md for the boundary map.

Stage 1 — Fast path (earn the right to skip the full method)

Most problems don't deserve the full apparatus. Try to win cheaply first — but under discipline, with a strict exit.

• Known-fault check: does the symptom match a known issue, recent advisory, or an error message with a well-documented cause? Search first; a recognized problem needs recognition, not analysis.
• Cheap discriminating tests, ordered by probability ÷ cost. Prefer the test that is most likely to hit, cheapest to run, and — best of all — splits the possibility space (a half-split beats ten one-at-a-time checks in a chain of components; see references/test-toolkit.md).
• One change per test. Vary one thing, observe, record. Shotgun changes destroy the evidence Stage 2 will need.
• Exit condition (count honestly): after ~3 failed hypotheses, or when the candidate space is clearly large/opaque, or when any single test is expensive, slow, risky, or the fault won't reproduce on demand — stop guessing and escalate to Stage 2. Log the failed attempts; they become IS-NOT data ("we swapped the PSU and nothing changed" is evidence).

Stage 2 — Specify

Build the comparative specification. This is the heart of the method.

Problem statement first: one object + one deviation, stated factually ("backup job nightly-pg exits 137 on 3 of 12 VMs since 07-19" — not "backups are broken"). Test it: if the statement is already explainable, back up to the thing that isn't. A brief why-chain on the symptom helps strip explainable layers before specifying.

The table. Four dimensions × IS / IS-NOT, plus what distinguishes them:

	IS	COULD BE, but IS NOT	Distinctions / Changes
WHAT — which object, which deviation
WHERE — where observed; where on the object
WHEN — first seen; since then (pattern?); when in lifecycle
EXTENT — how many objects; how bad; how many per object; trend

Rules that make the table work:

• IS-NOT is the closest logical comparison, not "everything else". The sibling VM that didn't fail, the identical node that stayed up, the time window when it didn't happen. The closer the comparison, the sharper the boundary around the cause.
• Ask every cell. An honestly-empty cell ("N/A" or "unknown — need to check") is fine; a skipped cell is a hole the favorite theory will hide in. Unknown cells are the data-collection shopping list.
• Tag provenance per hard rule 1.
• Then mine it: for each IS/IS-NOT pair ask what is distinctive about the IS side? — and for each distinction, what changed in, on, or around it, and when? Dated changes that line up with the WHEN row are prime cause material.

Full question set, worked example, and table-maintenance guidance: references/specification.md. When facts are in logs, compress them first (the lessence skill pairs well) and transcribe findings into cells with provenance.

With a human partner: request facts one dimension at a time — a focused question set they can actually answer — never a 20-question dump. You keep the table; they fetch the facts.

Stage 3 — Hypothesize

Generate 3–5 candidate causes in parallel — "candidate", "hypothesis", and H1..Hn all name the same thing here — each stated as a mechanism, not a component: not "the switch", but "switch port MTU dropped below tunnel overhead, so large packets silently fragment and the session stalls". A mechanism can be tested; a pointed finger cannot.

Sources, in order:

1. Knowledge and experience — cast a wide net from the agent's and the user's knowledge of systems like this. Fastest when it works.
2. Distinctions and changes — derive causes mechanically from the table when brainstorming yields nothing plausible, too much, or only candidates that fail Stage 4.

Parallel matters: a single cherished hypothesis is the documented anchor-trap. Keep a live numbered list (H1..Hn) with status (alive / refuted / verified) and tag every piece of evidence with the hypotheses it bears on.

State each mechanism at the weakest level the table forces. The spec sets a floor on specificity (the candidate must explain every IS and spare every IS-NOT) and testability keeps it refutable — but do not exceed that floor. Every detail not purchased by a table cell is an unforced commitment that makes the hypothesis strictly less probable while making the story feel more explanatory (the conjunction fallacy). "Something in the upgraded DB version raises peak memory" beats "the v16 parallel-dump default doubles work_mem" until a cell forces the narrower claim.

Edge cases change the move (see references/edge-cases.md): if the thing never worked ("Day One" deviation), there is no change to find — hunt distinctions against a working sibling only. If the decline was gradual, suspect drift and don't seize the most visible recent change. If the fault is intermittent, specify occurrences vs. non-occurrences over time.

Stage 4 — Paper test (kill candidates before spending on tests)

For each candidate, walk the full table: "If H3 is the cause, how does it explain each IS — and each IS-NOT?"

• A candidate that requires the IS-NOT side to be false is refuted by evidence already in hand. Kill it. This is free — no lab time spent.
• A candidate that fits only with extra assumptions survives provisionally; write the assumptions down. Rank survivors weakest first: the candidate that commits to the least beyond what the table forces. Assumptions and unforced detail are the same currency — specificity must be purchased by an IS or IS-NOT cell, and every unpurchased commitment is another way to be wrong when the next cell fills in. The weakest survivor is the one most likely to still fit the evidence you haven't collected yet.
• The top survivor is the most probable cause — a rank, not a verdict. Paper never proves; it only prunes.

If every candidate dies: hard rule 5 — the spec is missing a distinction or a change. Collect more facts; consider whether two problems are bundled under one name (back to Stage 0).

Stage 5 — Verify in the world

Confirm the most probable cause with the safest, surest, cheapest, fastest real-world check available:

• Observe the mechanism in action (capture the packet, watch the counter, catch the OOM kill in the log).
• Experiment: swap the suspect component, or better, reverse the suspected change and watch the problem stop — then, ideally, re-apply it and watch the problem return (the strongest evidence there is).
• Fix and monitor when direct observation is impossible — apply the fix that follows from the mechanism and watch the specific symptom, over a window long enough to be meaningful (for intermittent faults: several times the longest observed gap between occurrences).
• When the evidence is destroyed or unreachable, verify the assumptions the candidate depends on instead.

Physical/production actions belong to the human partner when one is present; the agent specifies exactly what to do and what result each outcome would imply; the human executes and reports. "If you didn't verify it, it isn't fixed" — a fix that happens to coincide with recovery, unverified, is how the same incident returns next month.

Close the loop: state the verified cause, the fix, and update the spec table one last time showing the cause explaining every row. If the analysis revealed contributing conditions worth systemic attention, note the handoff to a proper retrospective — don't do it here.

Escalation and red flags

• Escalate to the human after 3 failed verification attempts (not paper kills — those are progress), or when the surviving cause implies a design problem rather than a fault.
• Red flags that mean return to the method, spoken by either partner: "it's probably just X", "let's try reinstalling everything", "must be a <vendor> bug" (without a mechanism), proposing a second fix while the first is unverified, an IS-NOT cell contradicting a hypothesis everyone still likes, three swaps in a row with no new table entries, a hypothesis more specific than any cell forces (the vivid detailed story is the anchor-trap wearing its best clothes).
• Proportionality, restated: severity does not dictate ceremony. A severe problem with an obvious verified cause needs no table; a "minor" recurring annoyance that has eaten four debugging sessions deserves the full method.

References

• references/specification.md — full question set per dimension, worked example, provenance and table-maintenance discipline. Read when entering Stage 2.
• references/edge-cases.md — Day One deviations, gradual drift, intermittent faults, multi-causal/systemic boundaries, handoff map. Read when the standard change-hunt feels wrong.
• references/test-toolkit.md — test-selection math (probability ÷ cost), half-split, families-of-variation pruning, one-variable discipline, delta debugging, best-vs-worst comparison. Read when choosing what to test next.
• references/evidence.md — why each mechanism is in the skill, with the research behind it. Read when curious or when adapting the method.
• references/sources.md — dated per-URL index behind evidence.md; freshen passes stamp Last verified: there. No need to read during troubleshooting.