Record replay

Record & Replay — turn a screen workflow into a skill

This plugin captures what the user does on screen, has a vision model understand the workflow, and writes a reusable SKILL.md. It is a cross-platform, China-model-friendly port of Codex's Record & Replay.

How it captures

By default the recorder runs in video mode: it captures a continuous screen video AND records the user's mouse clicks + keystrokes as a timestamped event log. This means nothing is missed — a transient click or a typed value that falls between screenshots is still captured by the event recorder. At generate time the video plus the event timeline are sent to the vision model together, so it understands the exact actions, not just the resulting screen states.

• --mode video (default when ffmpeg is available) — full video + input events.
• --mode frames — periodic screenshots only (no ffmpeg needed).
• --fps N — video frame rate (default 5).
• --no-events / --mask-text — control input capture (see Privacy).

Privacy: keystrokes are recorded by default because typed values are needed to parameterize skills. Everything stays local and is sent only to the provider you configured. Use --mask-text (record positions and key names, mask typed characters) or --no-events for sensitive workflows.

The 3-step flow

1. Record — the user demonstrates the task on screen while a background process captures frames.

   /record-replay:record <name>
   

2. Stop — finalize the recording.

   /record-replay:stop <name>
   

3. Generate — select key frames, send them to the vision model, render a skill.

   /record-replay:generate <name>
   

Prefer guiding the user through these three commands in order. The recording name is a filesystem-safe slug (e.g. submit-expense-report).

One-time setup (first use)

Dependencies and an API key are needed. Check with:

orr doctor

• If a dependency is missing: pip install -r requirements.txt (mss, Pillow, requests, imageio-ffmpeg, pynput).
• Pick a vision provider and set a key. MiniMax M3 is the default:

  orr config set provider minimax
  orr config set providers.minimax.api_key <YOUR_KEY>
  

  Keys also work via env vars: MINIMAX_API_KEY, DASHSCOPE_API_KEY (Qwen), ZHIPU_API_KEY (GLM), OPENAI_API_KEY.
• To validate end-to-end without a key or network, use the offline mock: add --provider mock to generate.

Providers

Provider	Model	Endpoint	Env var
minimax (default)	MiniMax-M3	api.minimax.io/v1 (global) or api.minimaxi.com/v1 (China)	MINIMAX_API_KEY
qwen	qwen-vl-max	DashScope compatible-mode	DASHSCOPE_API_KEY
glm	glm-4v	Zhipu bigmodel	ZHIPU_API_KEY
openai	gpt-4o	OpenAI	OPENAI_API_KEY
mock	—	offline test	—

Switch the China endpoint for MiniMax with orr config set providers.minimax.base_url https://api.minimaxi.com/v1.

Without a live recording

If the user already has a video or a folder of screenshots:

orr frames <name> --from-video recording.mp4      # needs ffmpeg
orr frames <name> --from-dir ./screenshots
orr generate <name>

Where skills land

By default, generated skills are written to .claude/skills/<name>/SKILL.md in the current project (use --scope user for ~/.claude/skills/, or --out <path> for an exact file). Tell the user to reload skills/plugins so the new skill is picked up immediately.

Tips

• The recorder dedupes static frames and caps the frame count, so a long recording still yields a compact, focused set for the model.
• Encourage the user to perform the workflow at a steady pace and switch screens/windows deliberately so changes are captured.
• Generated skills are generalized (specific values become parameters), making them reusable — emphasize this value to the user.