Cohort pattern diagnostics

Cohort Pattern Diagnostics

The Idea

Do not explain a repeated anomaly until you know whether it is one phenomenon.

When many cases look similar from the outside, the first tempting move is to pick a representative example and go deep. That is sometimes right. It becomes dangerous when different failure classes have been collapsed into one syndrome: counterexamples from one class can falsely kill the right explanation for another class, and the investigation starts to feel impossible.

This skill changes the first question from "what caused it?" to:

Are these cases one population, or several populations wearing the same mask?

Once the cohorts are separated, use the normal domain method for each cohort: causal trace, code navigation, experiment, source review, proof step, or operational mitigation. The cohort pass is the gate that prevents mixed evidence from poisoning the root-cause search.

Fit Check

Use this skill when at least one of these is true:

• A repeated bug, crash, regression, report, result, or contradiction has no coherent single explanation.
• A single-case deep dive produces clues, but other examples keep breaking the story.
• Logs, stack traces, summaries, user reports, or metrics are too noisy to classify reliably from text alone.
• A rare event seems too unlikely, but it happens often enough to matter.
• The symptom appears across regions, machines, versions, scenes, users, sources, proof branches, or experiments in a way that may hide subgroups.
• A hypothesis explains some examples cleanly and others badly.

Let it go when:

• There is only one observed case and no way to gather comparators.
• A direct domain test already separates the likely causes.
• The issue is a tiny local mismatch with an obvious owner and no repeated population.

The Core Move

Use the lightest version that separates the evidence:

SYNDROME -> POPULATION -> FEATURES -> COHORTS
-> SAME-COHORT COUNTEREXAMPLES -> PER-COHORT CAUSE
-> RATE CHECK -> OBSERVABILITY UPGRADE -> RESIDUALS

1. Define The Syndrome

State the shared outside shape without smuggling in a cause.

Good:

The process crashes on return-like control transfer with corrupted stack state.

Risky:

The return address is being overwritten by our application code.

The syndrome is what cases have in common before explanation.

2. Build The Population

Gather all available cases or a deliberately bounded sample.

Record:

• population boundary: time range, environment, artifact set, source class;
• inclusion rule: why a case belongs;
• exclusion rule: why a similar case does not belong;
• data quality: complete, partial, biased, corrupted, stale, or unknown;
• known false positives and false negatives.

Prefer primary artifacts over summaries when classification is fragile: core dumps over crash text, traces over dashboard labels, raw logs over rollups, source records over generated summaries, direct user-path evidence over memory.

3. Extract Discriminating Features

Choose features that could plausibly split causes, not every available field.

Common feature families:

• time: start date, ramp, lifetime, before/after change, recurrence spacing;
• place: region, host, rack, machine, VM, scene, branch, dataset, source corpus;
• version: binary, dependency, kernel, compiler, model, package, content build;
• path: call path, user path, query path, task path, proof branch, workflow;
• load: rate, pressure, concurrency, memory, CPU, GPU, IO, batch size;
• shape: stack form, error form, corruption form, missing field, visual artifact;
• actor: user segment, agent role, service, subsystem, external provider;
• survivorship: first use, long-running nodes, restarted nodes, cached state;
• observer: which tool can see it, and which observer might be lying.

Keep feature extraction honest: an unavailable feature is unknown, not negative.

4. Split Before Explaining

Create a small cohort map. Start with two to five cohorts unless the evidence clearly needs more.

Each cohort needs:

cohort id / inclusion signal / excluded near-misses / size or count
/ distinguishing features / suspected cause class / confidence

Do not average across cohorts. A blended rate, blended stack, blended metric, or blended report can describe none of the real causes.

5. Test Counterexamples Inside The Same Cohort

A counterexample only attacks a hypothesis if it belongs to the same cohort.

Ask:

• Does this counterexample share the distinguishing features of the cohort?
• Could it belong to another cohort with the same syndrome?
• Is it a real counterexample, or a classification failure?
• Would moving it change the cohort boundary, the hypothesis, or both?

This is the main guardrail. Mixed-population counterexamples are often the reason a correct explanation looks impossible.

6. Diagnose Each Cohort Separately

For each cohort, switch from population thinking back to causal thinking:

cohort -> hypothesis -> mechanism -> expected signal
-> falsifier -> next test -> mitigation

If the cohort is code or infrastructure, identify the writer, live record, reader, decoder, cache/lifecycle, transform, invariant, and feedback path.

If the cohort is research, identify source class, claim boundary, conflicting evidence, falsifier, and the next external check.

If the cohort is UX, gameplay, or creative work, identify the user-visible surface, state path, perception trigger, and the contrast case.

7. Run The Rate And Plausibility Check

Rare mechanisms can be operationally common when the opportunity rate is high.

Estimate:

event probability per opportunity
* opportunities per second/minute/day
* affected hosts/users/tasks/scenes
= expected observed rate

Use rough order-of-magnitude math when exact data is unavailable. The question is not whether the estimate is beautiful; it is whether the mechanism is plausible enough to remain alive, or too small to explain the observed cohort.

8. Upgrade Future Observability

A fix that leaves the next recurrence invisible is incomplete.

For each meaningful cohort, ask what would make recurrence cheap to detect:

• a new field in logs;
• a stronger crash label;
• raw state captured before it is corrupted;
• a dashboard split by the discriminating feature;
• a runbook entry;
• a guardrail, denylist, health check, or quarantine path;
• a test fixture or reproducer;
• a source or claim ledger entry;
• a memory or skill update if the lesson generalizes.

The upgrade should reduce future diagnosis cost, not merely document the story.

9. Carry Residuals

Do not force unresolved cases into the winning explanation.

Residuals need:

case / why it does not fit / closest cohort / missing feature
/ next discriminating signal / risk if ignored

Residuals are not failure. They are the seed of the next cohort pass.

Output Shape

For a compact diagnosis, return:

Syndrome:
Population boundary:
Data quality:
Discriminating features:
Cohort map:
Same-cohort counterexamples:
Per-cohort hypotheses:
Rate/plausibility check:
Next decisive signal:
Mitigation or fix path:
Observability upgrade:
Residuals:
Confidence:

For code or operational work, include file paths, logs, metrics, commands, or runtime artifacts only where they prove classification or cause. For research, include sources and claim boundaries. For creative or UX work, include the real user path and contrast cases.

Using This As A Standalone Agent Skill

This skill has no required companion skill, tool, framework, or model. It can be used by any agent that can inspect evidence, compare cases, and keep uncertainty labels honest.

Use it as a first-pass diagnostic gate:

• Before root-cause analysis, ask whether the repeated syndrome hides several populations.
• Before causal tracing, separate cohorts so each path is traced inside one coherent group.
• Before code investigation, identify which runtime, version, machine, input, lifecycle, cache, or dependency cohort is actually affected.
• Before research synthesis, separate source classes, study designs, datasets, or claim families that may be producing incompatible evidence.
• Before UX, gameplay, product, or support analysis, split user-visible reports by the real path that produced them.

After the cohort map is stable enough, return to the best domain method available: debugger, experiment, source review, metric query, user-path replay, formal argument, incident response, or implementation fix.

What To Resist

• Do not diagnose the average of a mixed population.
• Do not treat the first deep case as representative until it has earned that role.
• Do not let counterexamples from one cohort kill a hypothesis for another.
• Do not classify from corrupt observers when stronger artifacts exist.
• Do not collect more data without naming which feature would split causes.
• Do not dismiss rare races, hardware faults, edge cases, or human-path effects until their opportunity rate has been checked.
• Do not call a fix complete if recurrence would still be hard to detect.
• Do not hide residual cases inside the nearest successful story.

Closing Check

Before finishing, ask:

• Is the syndrome defined without assuming the cause?
• Is the population boundary explicit?
• Are the cohort splits based on discriminating features rather than vibes?
• Were counterexamples checked against the same cohort?
• Does each cohort have its own hypothesis, falsifier, and next signal?
• Was rate plausibility tested for rare mechanisms?
• Did the fix or recommendation improve future observability?
• Are residuals named instead of smoothed away?

If any answer is unstable, shrink the population boundary, improve the feature set, or run one cheaper discriminating signal before claiming root cause.