Porcupine analyzer

Porcupine Failure Analyzer

Quick start

python3 SKILL_DIR/scripts/analyze_porcupine_failure.py path/to/violation.html

The script reads the HTML, finds non-linearizable partitions, and prints a diagnosis to stdout. The first line of each partition section is a one-line headline naming the contradicting op, the frontier state, and the most likely culprit — start there.

How to read the output

A typical diagnostic loop:

1. Read the headline. Each partition section opens with: DIAGNOSIS: op#K c<id> <description> contradicts frontier state X; most likely culprit: op#J c<id> <description> (token T, produces_frontier_state) This is your starting hypothesis. If the headline matches the failure you'd expect, you're often done at this step.

2. Check the frontier state. What does the model think was in place at the failure point? Is that what your test should have produced?

3. Read the blocked candidate's description. What did it return? Why does that contradict the frontier state?

4. Use value-token provenance to trace where the unexpected value came from. An overlap hit means a concurrent op produced it; a before hit means an older op wrote it and something should have invalidated it. The row tagged (..., produces_frontier_state) is the writer the model believed left the cache in its current state.

5. Open the HTML in a browser for the visual timeline. The analyzer has already told you the op# to focus on; the visualization places it in context with its concurrent peers.

Annotated example

Cache-style model (Insert/Get/Delete) where a Delete claims the key is missing but the model thinks an Insert of value 3049 should still be visible. Excerpt:

## Partition[0]

- history operations: 600
- distinct partial linearizations recorded: 1
- longest partial: length 457 (1 ties)

## Frontier 1 (partition 0)

DIAGNOSIS: op#456 c10 Delete() -> not found contradicts frontier state
{true 3049}; most likely culprit: op#455 c3 Insert(3049) -> ok
(token 3049, produces_frontier_state)

- state trajectory (last 5 of 457 steps):
  - op#451 c3 Delete() -> deleted (was 3043): {true 3043} -> {false 0}
  - op#452 c3 Insert(3046) -> ok: {false 0} -> {true 3046}
  - op#455 c3 Insert(3049) -> ok: {false 0} -> {true 3049}

- frontier state (last StateDescription): {true 3049}
- blocked candidates (real-time enabled, not in partial): 1
  - op#456 c10 [96458,96458] Delete() -> not found [out_partial]
                                              | frontier_state={true 3049}
    value-token provenance:
      - token 3049: before=1 overlap=0 after=2
        - op#455 c3 [96333,96416] Insert(3049) -> ok
                                          (before, produces_frontier_state)

The headline already tells the whole story: op#456's Delete missed a key that op#455 had just inserted. The value-token row confirms it: 3049 was introduced by op#455 (tagged produces_frontier_state because it's the last admitted step before the failure), and downstream ops (which fail with op#456) observe 3049 too.

CLI reference

Flag	Default	Meaning
--partition N	auto	Restrict to one partition index
--max-frontiers N	2	Number of longest-partial views per partition (≥ 1)
--max-blocked N	6	Blocked candidates printed per frontier (≥ 1)
--max-neighborhood N	15	Overlapping ops per blocked op (0 disables)
--max-tokens N	4	Value-token groups per blocked op (0 disables)
--max-sample N	5	Example ops shown under each value-token row (≥ 1)
--trajectory N	5	State transitions shown before each frontier (0 disables)
--max-desc-len N	160	Truncate printed descriptions to this many chars (≥ 1)
--client N	—	Display filter: only show ops from client N (repeatable)
--op-range A:B	—	Display filter: only show ops with id in [A, B]

The two display filters narrow what's PRINTED; they do NOT change which ops the analyzer considers when finding blocked candidates. Use them to focus the output on a specific client or time window after seeing the headline.

To analyze every violation HTML produced by a test run (portable across bash without shopt -s globstar and zsh):

find tests \( -name '*violation*.html' -o -name 'porcupine_*.html' \) -print0 |
while IFS= read -r -d '' f; do
  echo "=== $f ==="
  python3 SKILL_DIR/scripts/analyze_porcupine_failure.py "$f"
done

What each section means

The analyzer is model-agnostic. It cannot tell you "your Delete returned the wrong sentinel" — that classification requires reproducing the model. Instead it surfaces the raw evidence the model's own classifier would need, in a form that's actionable for every Porcupine model.

Failure Digest

Per-partition summary including peak_concurrency, linearized/history ratio, longest partial length, and cascade_size. A large cascade size (>20% of history) indicates the violation fires early and downstream ops inherit the inconsistency; a small cascade size indicates a localised contradiction.

Operation Mix

Counts ops by the prefix of their model-rendered description (whatever sits before the first (). Useful for spotting workload skew without knowing the op vocabulary in advance. Scoped to the partition(s) you're analysing.

Annotations

If your test code emitted Porcupine Annotation markers (e.g., barriers, client-state changes), they appear here. Each is also correlated per blocked op (see "Blocked candidates" below).

Partition section

Opens with the one-line DIAGNOSIS headline. Every op-summary line has the format:

op#<id> c<client_id> [<call_ns>,<return_ns>] <description> [<tag>] | <extras>

where <call_ns>/<return_ns> are the true ns timestamps (Porcupine's OriginalStart/OriginalEnd), <description> is whatever your model's DescribeOperation returned, <tag> is in_partial or out_partial relative to the current frontier (NOT "in any partial"), and <extras> carry the frontier state and any other contextual annotations.

State trajectory — the last N admitted steps, each line is one op's effect formatted as op#K cN <description>: <prev-state> -> <next-state>, with (no state change) for read-only steps. Read top-to-bottom to see how the model believes the state evolved up to the failure point.

Frontier state — the model's state at the end of the longest partial linearization. The blocked op was rejected against THIS state. If the description is Get(K) -> 5 and the frontier state is value=10, you know the Get's return contradicted what the model believed was current.

Blocked candidates (a HEURISTIC, not an authoritative list) — operations not in the longest partial whose real-time predecessors are all in the partial. The script reports candidates Porcupine could have considered extending the partial with; whether Porcupine actually enumerated each one depends on its search strategy, which the HTML doesn't carry. Treat as "candidate culprits." Each is printed with its description and the frontier state for direct comparison, plus any annotations whose time window overlaps [call_ns, return_ns].

Temporal neighborhood — every other op overlapping the blocked op's window, each tagged in_partial / out_partial. Shows which concurrent operations could have interfered. Look for out_partial ops on the same key/value as the blocked op — those are race candidates.

Value-token provenance — model-agnostic token matching. Pulls numeric tokens (42, 3.14, -7), quoted-string tokens ('A', "foo"), and identifier tokens (hot, Get) from BOTH the blocked op's description AND the frontier state's description, then finds every other op whose pre-extracted tokens contain each. State-side mining is what makes this work for blocked ops with no values of their own (Delete() -> not found has nothing to chase, but the frontier state {true 3049} does — mining 3049 from the state surfaces "op#K Insert(3049)" as the smoking gun).

Each row is bucketed into before / overlap / after the blocked op's time window. The row tagged (..., produces_frontier_state) is the op whose effect IS the frontier state — the most direct candidate culprit.

Ranking is rarest-first (hits-ascending): rare tokens are the most diagnostic. Tokens that index the SAME set of ops are coalesced into one row labeled {tok1, tok2} — e.g., found and not always co-occur in Delete() -> not found. Identifier-only groups dominated by a more diagnostic numeric or quoted group (4× hit-count) are suppressed with a "(suppressed N hits)" note — see the dominated-suppression heuristic at work in cases where partition keys swamp value tokens.

Identifier tokens are filtered two ways before ranking:

• Op-name filter: any identifier matching an op-mix prefix (i.e., it IS an op name like Get, Set, GetOrLoad) is dropped — structural noise, not a value.
• Partition-key fallback: any other identifier whose hit count exceeds 50% of the partition's history is dropped.

Numeric and quoted-string tokens are NEVER dropped — they're values.

Partial coverage (only when a partition has multiple partials AND some op appears in more than one) — for each blocked op, shows which OTHER partials admit it and at what lengths. Helps identify "this op is consistent with several shorter linearizations but doesn't fit the longest one."

What "blocked" means

Porcupine searches for an admission order of the entire history that's consistent with both real-time order and the model's Step function. When it gives up, the longest partial it found is the deepest prefix of that admission order it could prove.

The analyzer's "blocked candidates" are operations enabled by real-time order at that frontier (all their predecessors are already admitted) but not in the partial. This is a heuristic over-approximation: Porcupine may not have specifically tried every one — its search strategy isn't in the HTML. What you can say definitively: each candidate's observed output is inconsistent with the model's frontier state under the trivial sequential schedule "frontier ops in order, then this candidate."

Distinguishing Illegal from Unknown

The HTML doesn't carry Porcupine's verdict. A non-linearizable digest may reflect either a true violation (Illegal) or a checker timeout (Unknown). The analyzer prints an advisory note on every failure run, and the per-partition digest line includes a (no partials recorded — possible timeout/Unknown) annotation when the partition has zero partials.

Rule of thumb: if linearized/history < 0.10, the checker likely gave up early — re-run with a longer Porcupine timeout to rule out Unknown before chasing a phantom bug.

Conventions

• Don't commit test-run violation HTML files to your repo. Configure your test setup to write them under a path covered by .gitignore (e.g., tests/**/*.html).
• Display filters are display-only. --client and --op-range narrow the printed output; they do NOT change which ops the analyzer considers when finding blocked candidates. Run with no filter first to get the full diagnosis; add filters to focus follow-up reading.