Bit exact sim validation

Bit-Exact Simulator Validation

When you've ported a deterministic external system (a game engine, a PRNG, a settlement model, a protocol decoder) and want to know your port is correct, the gold standard is bit-exactness against captured authoritative state. This skill is the discipline: one trust-anchor test that asserts the port matches the original on a representative captured session, plus a per-step diff harness for when that test fails.

The work this saves is enormous: most porting bugs are silent — your port produces "plausible-looking" output that differs from the original in subtle ways and you discover months later when downstream code drifts. Bit-exact validation catches those at port time.

When this skill applies

• The system you're porting is deterministic (same input + same state → same output, always).
• You can capture authoritative state from the original — wire traffic, log files, recorded sessions, anything that gives you ground truth.
• You're going to use the port downstream — there's no point if you'd accept "approximately right".

If the system is non-deterministic (involves network randomness, timing, user input) you need a different validation approach (statistical equivalence, fuzz testing) — not this skill.

The pattern

The trust anchor test

A single test that loads a captured session, replays it through your port, and asserts the port matches the original output exactly. Pseudocode:

def test_trust_anchor():
    """If this fails, the port is wrong and everything downstream is unreliable."""
    captured = load_captured_session("fixtures/canonical-session.jsonl")
    result = replay_through_port(captured.inputs)
    assert result.final_output == captured.final_authoritative_output
    # Also assert per-step state where available
    for i, observed in enumerate(captured.per_step_states):
        assert result.per_step_states[i] == observed, (
            f"divergence at step {i}: port produced {result.per_step_states[i]!r}, "
            f"original produced {observed!r}"
        )

This test is your first line of defence. It's not a unit test; it's an integration test against captured ground truth. If it passes, downstream is trustworthy. If it fails, you have a bug; nothing downstream is reliable until you fix it.

The per-step diff harness

When the trust anchor fails, you need to know where it diverged. A diff harness walks the captured session step-by-step and prints the first divergence with full context:

def diff_against_capture(captured):
    """Walk every step; print the first state mismatch with surrounding context."""
    state = port_initial_state(captured.seed)
    for i, (action, observed_after) in enumerate(zip(captured.actions, captured.states)):
        port_after = port_apply(state, action)
        if not deep_equal(port_after, observed_after):
            print(f"DIVERGENCE at step {i}")
            print(f"  action: {action!r}")
            print(f"  pre-state:")
            print_state(state)
            print(f"  port produced:")
            print_state(port_after)
            print(f"  original produced:")
            print_state(observed_after)
            print(f"  diff:")
            print_diff(port_after, observed_after)
            return i
        state = observed_after  # advance using authoritative state
    return None  # passed

Key design: advance using the authoritative state, not the port's output. That way one bug at step N doesn't cascade into apparent bugs at steps N+1, N+2... You'll see exactly one divergence; the rest of the trajectory is faithfully measured.

Workflow

Phase 1: Capture authoritative state

Build a way to record ground truth from the original system. Sources:

• Wire traffic — if the system is a service you can talk to. Use android-mitm-setup for mobile apps, or browser DevTools for web. Save full sessions with mitmdump -w session.flows.
• Log files — if you can configure the original system to log per-step state.
• Driven sessions — a WS client or scripted browser that drives the system through a known sequence and captures everything (see closed-loop-live-demo for the WS pattern).

What to capture per session:

• Initial inputs: seed, config, starting state.
• Sequence of actions: every input that mutated state.
• Per-step state: after each action, the full state. (Or whatever subset you can observe — final state + intermediate cumulative outputs is enough to start.)
• Final output: the score, the result, the cumulative score, whatever the trust-anchor compares.

Save as one file per session, in a stable format (JSONL with one event per line works well).

Phase 2: Write the trust anchor test

The test goes in your test suite. It MUST be green before any other test you write is meaningful.

# tests/test_trust_anchor.py
def test_validate_against_canonical_capture():
    """Load fixtures/canonical-session.jsonl, replay through port, assert
    output exactly matches the captured authoritative output. If this is
    red, fix it before believing any other test in this module."""
    fixture = REPO_ROOT / "fixtures" / "canonical-session.jsonl"
    report = validate_against_capture(fixture)
    assert report.passed, report.first_divergence_description
    assert report.final_score_port == report.final_score_authoritative

When the test is green, you have an evidence-backed claim that the port reproduces the original on at least one representative session. Capture more sessions to broaden coverage.

Phase 3: Build the diff harness

The diff harness is a standalone script — scripts/play_and_diff.py is a good name — that:

1. Connects to the original (or loads a fixture).
2. Drives a session, capturing per-step state.
3. Replays the same inputs through the port.
4. Compares step-by-step, prints the first divergence with rich context.

The script's output is the input to debugging. Make it verbose: full pre-state, full post-state-of-port, full post-state-of-original, structured diff.

# scripts/play_and_diff.py — invocation shape
async def main():
    # Connect, drive, capture
    captured = await play_and_capture(target_config)
    save_json(captured, f"scratch/play-and-diff-{ts}.json")

    # Diff against port
    first_divergence = diff_against_port(captured)
    if first_divergence is None:
        print("✓ NO DIVERGENCES — port agrees with original every step")
    else:
        print(f"✗ Divergence at step {first_divergence}")

Phase 4: Iterate to convergence

When the diff finds a divergence:

1. Look at the first divergence only. Don't try to interpret later steps — they're meaningless until step N is fixed.
2. The shape of the divergence often points at the bug class. See references/divergence-signatures.md.
3. Fix the bug in your port. Re-run the diff. Either you find the next divergence (closer to the end) or you reach 0 divergences.
4. When you reach 0 divergences on one session, capture another and repeat. Different sessions exercise different code paths.

When several independent sessions all show 0 divergences across hundreds of state-mutations, the port is trustworthy.

The trust anchor: lessons

Empirical observation through a noisy channel isn't ground truth

A common failure mode: you "validated" your port by watching it run alongside the original and noting they "look the same" by eye. They don't look the same — your eye is doing massive data reduction. Real validation requires structured per-cell comparison.

A canonical example from one project: a rule about when a special tile spawns was learned by observing pixel animations in screen captures. The "rule" had ~41% observed accuracy — but the true rule had no such constraint, the 41% was classifier noise. The wrong rule survived for months because the validation channel (eyeballing animation) couldn't distinguish it from the right rule. Once the WS-script diff (skill E + this one) was built, the actual rule fell out in one session.

The lesson: build the bit-exact channel. Eyeballing is not validation.

The right tool sometimes already exists in the repo from earlier work

The same project's WS client was originally built for security recon — to send malformed requests at the server and observe responses. Months later, when sim-vs-server bit-exactness became the bottleneck, that WS client was the exact piece needed. The lookup that worked was "do I have a way to read authoritative per-step state?" — not "do I have a sim-divergence debug tool?"

When stuck on validation, look across your repo for any code that talks to the live system. Often there's reusable infrastructure.

Optimisation can make the underlying bug visible

If your port has a rare bug that triggers only on edge-case states, you may not hit it until you run many sessions. Optimising the port to run fast enough that you can drive 10 sessions per hour will surface the bug that "never happens".

The flip side: the bug was there the whole time; optimising just made it observable. "Make it fast" and "find the rare bug" are sometimes the same project.

Validate before fixing

When something looks broken — "my port doesn't match production" — your first instinct will be to debug the port. Resist it. First, run the diff harness and confirm the port IS wrong. Sometimes the symptom you noticed has a different cause (a CV race, a stale capture, a timing issue). 15 minutes of diff-running can save hours of chasing the wrong bug.

Divergence signatures

The shape of the divergence often tells you the bug class. See references/divergence-signatures.md for examples. As a teaser:

• Many scattered wrong cells → bug in the state-mutation rule itself (mark detection, scoring, etc.).
• Top-of-one-column wrong, refilled values only → bug in the PRNG refill order, or a CV / timing race.
• One bit consistently wrong across all cells → endianness or sign-extension bug.
• First step matches, second step wrong → PRNG state advanced by wrong amount.
• Final score wrong by an integer multiple → scoring multiplier off by one.

Common pitfalls

• The trust anchor test passes, but downstream code still misbehaves. The captured session doesn't exercise the edge case your downstream code hits. Capture more sessions, including ones that exercise unusual paths.
• The diff harness shows different divergences each run. Either the original isn't actually deterministic (recheck), or your port has uninitialised state (check for default = None slots).
• You "fix" a divergence and a new one appears 5 steps later. Two interacting bugs — common when you've been guessing fixes rather than reading the divergence shape. Slow down; read the diff.
• Captures contain sensitive data. Sessions include tokens, account IDs, sometimes PII. Don't commit raw captures to a public repo; commit redacted minimal fixtures.

References

• references/divergence-signatures.md — patterns in the shape of wrong state and what they typically mean.

Pairs with

• android-mitm-setup — capture authoritative state from mobile traffic.
• closed-loop-live-demo — same WS infrastructure that drives the live demo also captures sessions for this skill.
• time-window-seed-bruteforce — if the system is seeded and the port doesn't match initial state, your seed recovery may be wrong.
• negative-result-branches — when you can't fix a divergence, document the attempt on a branch.