Record and replay skill

Record & Replay (browser + desktop)

Record & Replay lets an agent learn a user-demonstrated workflow and turn it into a reusable skill. Two modes:

• Browser mode (default, richest evidence): a headed Playwright browser (system Google Chrome if available) records every user action into events.jsonl with multiple selector candidates, plus Playwright trace chunks (trace-NNN.zip). Works on macOS, Windows and Linux.
• Desktop mode (native apps): a lightweight single-process recorder (pynput) captures global mouse/keyboard events, active-window changes and click-moment screenshots into desktop-events.jsonl + screens/ — Codex-parity (Codex records an event/context stream too, not video), ~1-3% CPU. Works on macOS, Windows and Linux (X11). See "Desktop recording" below.

Pick browser mode whenever the workflow is browser-only — its semantic selectors produce far better skills than desktop coordinates.

Setup (once)

cd ~/.agents/skills/record-and-replay
node setup.mjs      # npm install + (Chrome yoksa) Chromium + self-test
node install.mjs    # ajan skill dizinlerine bağla (macOS/Linux symlink, Windows junction)

Installs the playwright npm package. Browsers are NOT downloaded when Google Chrome is installed (the recorder uses channel: 'chrome'); otherwise the script downloads Chromium. Both scripts are cross-platform (Node).

Recording Workflow

1. Start the recorder detached in the background only when the user is ready to begin. Always pass --name <slug> so the output directory is known up front:

   cd ~/.agents/skills/record-and-replay
   # macOS / Linux:
   nohup node scripts/recorder.mjs --minutes 30 --name <slug> [--url <startUrl>] \
     > ~/.agents/recordings/recorder.log 2>&1 &
   # Windows (PowerShell):
   Start-Process node -WindowStyle Hidden -ArgumentList "scripts/recorder.mjs","--minutes","30","--name","<slug>"
   

   Agents with a native background-run facility (e.g. Claude Code's Bash run_in_background) may use it instead. Outputs go to ~/.agents/recordings/<slug>/.

2. Verify startup once (~2-3 seconds after launching — a single check, not a polling loop): confirm ~/.agents/recordings/<slug>/session.json exists with "status": "recording" (the recorder also prints a {"status":"recording",...} line to the log on success). If the check fails, read the log: recording_already_active (exit 2) means another recording holds the lock; launch_failed or a missing-module error means setup is needed. Report the actual problem instead of telling the user recording has begun.

3. After the startup check, do not sleep, poll, or wait in a loop. End your turn: tell the user recording has begun, what the time limit is, and that a small floating control window (bottom-right, like Codex's recording controls) has a timer plus Pause/Resume, Stop and Discard (tap twice to confirm) buttons. Nothing is captured while paused — events and trace both stop. If they close the control window by accident it reopens itself; closing the main browser window also ends the session (avoid quitting the browser entirely with Cmd+Q, which can lose the last minute of trace). Pass --badge at start to use an in-page badge instead of the control window (fallback for exotic environments).

4. Pause/resume (user asks you to pause, or you need to skip a sensitive stretch): touch ~/.agents/recordings/<slug>/PAUSE pauses within 2 seconds; delete the file to resume. The control-window buttons and the PAUSE file stay in sync; session.json carries the current paused state. Paused intervals appear as paused/resumed markers in events.jsonl and are absent from the trace.

5. When the user says they are done:

   • If the process is still running, stop it gracefully: touch ~/.agents/recordings/<slug>/STOP (the recorder notices within 2 seconds and finalizes).
   • Read session.json — status: "recording" means still live; "completed" means outputs are final; "discarded" means the user pressed Discard — the evidence files were deleted, acknowledge and NEVER create a skill from it. endReason is one of user_stopped, user_discarded, browser_closed, stop_file, time_limit, signal, event_limit.
   • Inspect the captured events before responding: run node scripts/summarize.mjs <outDir> for a compact timeline, then read events.jsonl directly for steps that need detail.

6. Cancellation is defined by content, not endReason: if the recording contains few or no kind: "dom" events — whatever the endReason — treat it as cancelled. Acknowledge, do not create or update a skill.

7. If the recording contains enough information to identify a reusable workflow, create or refine a skill for it by default — even if the user did not explicitly ask. Do not stop after producing only a summary or a replay plan.

8. If the recording is too thin or ambiguous to identify the workflow, do not guess. Explain what is unclear and ask the user for the missing pieces.

Concurrent recording

Only one recording can be active at a time (lock file ~/.agents/recordings/.lock). If the recorder exits with recording_already_active, do not force-start another; tell the user a recording is already in progress and ask whether to use it or stop it first. Restarting with the same --name is safe: the previous events.jsonl is preserved as a timestamped .bak, not appended to.

Interpreting Events

events.jsonl is the primary evidence; session.json gives timing/paths only; the trace-NNN.zip chunks are the DOM-level ground truth (npx playwright show-trace <outDir>/trace-*.zip; session.json's traceChunks lists them in order). Trace defaults to light mode — full DOM snapshots per action but no continuous screenshots (keeps CPU low). Start with --trace full when the user wants pixel-perfect visual evidence, or --trace off for the absolute minimum footprint.

Each line is one event: {seq, t (ms since start), kind, type, url, ...}.

• kind: "page" — navigations, downloads, dialogs, tab opens/closes. Navigations segment the workflow into stages.
• kind: "dom" — user actions. target.selectors carries multiple selector candidates: testId, role ({role, name}), id, nameAttr, text, css. Prefer them in that order when generating replay code; css is the last resort.
• type: "input" events are debounced — one event per field with the final value (flushed on blur, Enter, change, and form submit). Password/credit-card/OTP-like fields (English and Turkish names: password, parola, şifre, kart, kimlik…) arrive pre-masked (***MASKED***, masked: true).
• selection and copy events reveal what content the user was working with — often the best clue when the user refers to "this" or "the thing I selected". Secret-looking selections (OTPs, API keys, tokens) are masked. paste events carry only length/word-count, never content.
• Sensitive data: masking is heuristic, not a guarantee — and it applies only to events.jsonl. The trace chunks are NOT masked: DOM snapshots contain raw input values (including values masked in the JSONL) and screenshots capture everything visible on screen (revealed tokens, 2FA codes, balances). Treat the entire ~/.agents/recordings/<name>/ directory as sensitive: inspect it locally only, never upload or commit it, and never quote literal secrets from it in summaries or generated skills — use placeholders or generic descriptions.

Creating Skills from a Recording

Before creating or refining a skill, read and follow the skill-creator skill if it is available in your environment. Create an actual discoverable skill (a SKILL.md folder), not just a Markdown runbook — and complete validation before reporting done.

• Treat the recording as evidence of the user's intended outcome, not a script to replay pixel by pixel. If a connector/API/CLI/MCP tool can achieve the same outcome semantically (create a calendar event, open a PR, post a message), prefer it over UI automation.
• When UI replay is genuinely needed, generate a Playwright script from scripts/replay-template.mjs: semantic locators (getByTestId > getByRole > getByLabel > getByText), a verification step after each critical action, secrets from env vars. Never use raw coordinates unless the event stream offers no better target.
• The recorder's persistent profile (~/.agents/recordings/.browser-profile) keeps logins across sessions, so generated skills usually need no login steps. Security: that directory holds live authenticated session material (cookies/localStorage; Playwright launches Chrome with a mock keychain, so treat it as plaintext credentials). Never commit, share, or back it up unencrypted — and remember that any replay skill reusing it runs with the user's logins. For recordings on untrusted or high-sensitivity sites, prefer a disposable profile (--out plus a temp profile) or clear the profile afterwards.
• Save new skills to ~/.agents/skills/<skill-name>/ and symlink them into the agent skill directories (see install.sh for the pattern) so every agent on this machine can use them.
• Parametrize what varies (dates, amounts, names) and hard-code what does not. State the workflow's purpose and verification criteria in the generated SKILL.md.

Desktop recording (macOS, Windows, Linux/X11)

Setup: node setup.mjs (creates the minimal .venv-desktop Python env — pynput+mss+pillow, ~50 MB; requires uv). Permissions: on macOS grant Input Monitoring + Screen Recording + Accessibility to the terminal app running the agent (System Settings → Privacy & Security). Without permissions the capture silently records 0 events — check with status after the first 10 seconds.

The engine is a single-process recorder (~0-3% CPU): input event stream + active-window changes + a half-resolution JPEG at each click (max 1/700ms) → desktop-events.jsonl + screens/. This is Codex-parity: Codex's Record & Replay also records an event/context stream, not video — do not try to add continuous video/screen capture; it can overwhelm the machine.

Workflow mirrors browser mode (all commands cross-platform):

1. Start (background is handled by the script): node scripts/desktop-record.mjs start <slug> [minutes] — outputs to ~/.agents/recordings/<slug>/. It prints {"status":"recording",...} on success; on recording_already_active or start_failed, report the real problem.
2. Verify once ~10s in: node scripts/desktop-record.mjs status <slug> — if the user is active but actionEvents stays 0, permissions are missing; tell the user which ones. Then end your turn as in browser mode (no polling).
3. Pause/resume: node scripts/desktop-record.mjs pause <slug> / resume <slug>. Pause closes the current segment and stops all capture; resume opens a new segment. status reports paused: true.
4. Stop: node scripts/desktop-record.mjs stop <slug> — signal-free on every platform: the recorder watches a STOP file and finalizes gracefully (also auto-stops at the time limit; works from paused state too). cleanStop: false in the output means the fallback kill was needed.
5. Inspect: python3 scripts/desktop-summarize.py <outDir> (Windows: python) → markdown timeline (window-titled sections, coalesced typing, clicks with their screenshot refs; segments merged in order with pause markers). Read a few JPEGs from screens/ around the key moments for visual context.
6. Then follow the same rules as browser mode: cancellation by content, create a skill by default, ask when ambiguous.

Desktop-specific caveats:

• NO masking: every keystroke and screenshot is raw. Treat desktop-events.jsonl and screens/ as fully sensitive; never quote literal secrets — use placeholders.
• Desktop events are coordinates + window titles, weaker than browser selectors. When generating a replay skill, map actions to the shared computer-use skill's semantic tools (accessibility-tree find + click/type via cua-driver) instead of raw coordinates; use the recording as intent evidence.
• One desktop recording at a time (~/.agents/recordings/.desktop-lock); browser and desktop recordings may run together only if the workflow genuinely spans both.

Files

File	Purpose
scripts/recorder.mjs	Browser recorder. --self-test runs a headless end-to-end check (throwaway profile).
scripts/summarize.mjs	events.jsonl → compact markdown timeline (--full for everything).
scripts/replay-template.mjs	Skeleton for generated browser replay scripts.
scripts/desktop-record.mjs	Desktop recorder driver: start/pause/resume/stop/status <slug> — cross-platform.
scripts/desktop-lite-recorder.py	Desktop engine: single-process event/window/click-screenshot stream (~0-3% CPU).
scripts/desktop-summarize.py	desktop-events.jsonl → markdown timeline (segments merged).
setup.mjs / install.mjs	Cross-platform setup / link into agent skill dirs (.sh wrappers exist for POSIX).