Closed loop live demo

Closed-Loop Live Demo

After you've ported a deterministic system and validated it bit-exactly against captured sessions (bit-exact-sim-validation), the next-level claim is: "given a fresh session, my port can predict the live service's outcome before the session even runs." This skill is the single command that proves it.

The framing matters. Offline benchmarks always look good. Captured-session replays look good. Predicting a live session you've never seen, in advance, end-to-end, in one command — that's the demonstration that forces honesty.

When this skill applies

• You have a port of a deterministic external system.
• The system exposes a way to start a session that you can drive programmatically (HTTPS REST + WebSocket is the common shape).
• The system ships enough state to the client at session-start that your port can predict the outcome (seed + config + ruleset).
• You can observe the actual outcome from the wire (cumulative score in clear msgpack, final-state message, etc.).

If the system requires real-time user interaction that's hard to script, the closed-loop demo is harder to build but the principle is the same: drive it deterministically, capture the outcome, compare.

The shape

One Python entry point. Inside it, the canonical sequence:

1. matchmake_post(entry_token) -> reservation
2. ws_connect(reservation.url)
3. handshake(ws)
4. send "request prepare" frame
5. recv "prepare" frame -> extract seed, config, ruleset
6. predicted = run_local_port(seed, config, planned_actions)
7. submit planned_actions (one-by-one or as a batch)
8. tail every state-mutation event; record cumulative score from each
9. recv "game over" frame or wait for the configured duration
10. actual = last_observed_cumulative_score
11. assert actual == predicted (or print delta)
12. write proof artifact: {seed, predicted, actual, n_events, match: "EXACT" | "Δ=..."}

The artifact is a small JSON file with a timestamp. Commit a few of them to your repo as evidence — they're tiny and they're the receipts.

Example skeleton

import asyncio, json, time
from pathlib import Path
from curl_cffi import requests as cffi
from websockets.asyncio.client import connect as ws_connect

async def closed_loop_demo(launch_url):
    """Single command. Talks to the live service, predicts, drives, compares."""

    # 1. Matchmake HTTPS POST to reserve a session
    entry_token = extract_token_from_launch_url(launch_url)
    reservation = matchmake(entry_token)  # POST → JSON

    # 2-3. WebSocket connect + protocol handshake
    async with ws_connect(reservation["ws_url"], compression=None) as ws:
        await handshake(ws, reservation["session_id"])

        # 4-5. Request initial state, parse seed
        await send_frame(ws, "REQUEST_PREPARE")
        prepare = await recv_frame(ws, type="PREPARE")
        seed = prepare["seed"]
        config = prepare["config"]

        # 6. Run local port
        actions, predicted_outcome = predict(seed, config, max_actions=30)

        # 7. Submit actions (one at a time, or as a batch — both work)
        for action in actions:
            await send_frame(ws, "ACTION", action)

        # 8. Tail every state-mutation event; record cumulative score
        events = []
        deadline = time.monotonic() + 120.0  # 2-min budget
        while time.monotonic() < deadline:
            try:
                ev = await asyncio.wait_for(recv_any_frame(ws), timeout=2.0)
            except asyncio.TimeoutError:
                break
            if ev["type"] == "STATE_UPDATE":
                events.append(ev)
            if ev["type"] == "GAME_OVER":
                break

        actual_outcome = events[-1]["cumulative_score"] if events else None

    # 9-11. Compare
    match = "EXACT" if actual_outcome == predicted_outcome else f"Δ={actual_outcome - predicted_outcome}"
    print(f"  seed:             {seed}")
    print(f"  predicted:        {predicted_outcome:,}")
    print(f"  actual:           {actual_outcome:,}")
    print(f"  events observed:  {len(events)}")
    print(f"  match:            {match}")

    # 12. Write proof artifact
    artifact = {
        "seed": seed,
        "predicted_score": predicted_outcome,
        "actual_score": actual_outcome,
        "event_count": len(events),
        "match": match,
        "timestamp": time.time(),
    }
    out = Path(f"recon/closed-loop-proof-{int(time.time())}.json")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(artifact, indent=2))
    print(f"  proof artifact written to: {out}")


if __name__ == "__main__":
    import sys
    asyncio.run(closed_loop_demo(sys.argv[1]))

Invocation:

python -m recon.closed_loop_demo "https://launch.example.com/play/<token>"

What "EXACT" looks like

$ python -m recon.closed_loop_demo "https://launch.example.com/play/abc123"
  seed:             1780433928329
  predicted:        58300
  actual:           58300
  events observed:  45
  match:            EXACT
  proof artifact written to: recon/closed-loop-proof-1780434060.json

That's the receipt: a fresh session, a never-before-seen seed, an outcome predicted from the seed alone, an actual outcome from the live service, matching exactly across 45 state-mutation events.

What "near-exact" looks like

  seed:             1780434360614
  predicted:        107800
  actual:           107500
  events observed:  81
  match:            Δ=-300

A delta means your port and the live system disagree about one specific event. Often it's a single rule that misfires in a rare configuration. Feed the session into your diff harness (bit-exact-sim-validation) to localise.

Practical notes

WebSocket library choice

websockets (the Python library) is the standard. Two settings worth knowing:

• compression=None — many production WS servers don't negotiate permessage-deflate; if you don't disable client-side, you get protocol errors mid-session.
• max_size=None or max_size=16*1024*1024 — server frames can exceed the default 1MB limit on busy sessions.

Origin and Referer

The matchmake endpoint and the WS upgrade often check Origin and Referer. Set these to whatever the production page uses (you can read it from your mitm capture or DevTools' Network panel). Mismatch → 502 from the upgrade.

TLS-fingerprint WAF on the matchmake POST

The matchmake endpoint is often WAF'd. Use curl_cffi with impersonate="chrome" for that HTTP POST — see mobile-auth-replay for the full pattern. The WS upgrade itself is typically NOT WAF'd because WAFs that inspect handshakes can't reliably parse WS upgrade negotiation.

Cumulative score on the wire

In many protocols (Colyseus, custom binary, msgpack-over-WS), the cumulative score lives in plaintext on per-step state-mutation events, even when the room state itself is schema-encoded. Reading the score doesn't require implementing the schema decoder — just decode the relevant msgpack field on each state event.

Entry-token validation may be lax

Surprisingly often, the matchmake endpoint accepts entry tokens it shouldn't:

• 64-zero placeholder strings work.
• Same token can be replayed many times.
• Tokens issued for one user/session work for arbitrary other users.

This is distinct from the launch-page endpoint which is usually stricter. If you find your matchmake call works with a known-invalid token, that's worth a private disclosure to the operator.

Re-usable launch URLs

In some implementations, a launch URL keeps reserving fresh sessions long after the original "consume" event. You can re-fire the same URL repeatedly for testing — saves credentials and avoids polluting your account state. Confirm against the specific system before relying on this; it varies by operator.

Why this matters more than offline benchmarks

Offline:

• Captured session + my port = match. ✓ (But the capture is what it is — I didn't predict it, I replayed it.)
• Greedy beats Random on synthetic seeds. ✓ (Both are running in my own simulator. Of course they agree.)

Live closed-loop:

• Fresh seed I've never seen, predicted offline, then the live service confirms in real time. ✗ if my port is wrong, ✓ if it's right.

A green offline test only proves your port is consistent. A green live closed-loop test proves it's correct against reality.

This is the demo to lead with when explaining the project to someone. Tables of offline numbers don't move people. A single live command that says EXACT does.

Disclosure implications

A closed-loop demo against a live service can demonstrate exploitability of an information-disclosure issue (e.g. "the system leaks the seed, and given the seed I can predict the outcome exactly"). When demonstrating to the operator:

• Send the proof artifact JSON, not the running command.
• Redact your specific session token / launch URL.
• Note that the demo runs as your own user against your own session; it doesn't impersonate anyone or read other users' state.
• Offer to retract the public demo if the operator wants disclosure-window time to deploy a fix.

This skill produces evidence; how you use the evidence is the disclosure question, not a technique question.

Pairs with

• bit-exact-sim-validation — the port that produces predicted must be validated first.
• mobile-auth-replay — if the matchmake endpoint requires an auth bearer, the auth-replay helper gets you the token.
• android-mitm-setup — observe the actual matchmake / WS protocol before scripting it.
• time-window-seed-bruteforce — if the seed isn't shipped on the wire, you may need to brute-force it from observed state.