Cua computer use

Skill alexleekt/agents/skills/cua-computer-use

AI agent skills and configurations managed by Saddle. Centralized behavioral rules for Claude, OpenCode, Codex, Cursor, Copilot, Gemini, and Pi.

Install

npx -y skills add alexleekt/agents --skill cua-computer-use

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

**ALWAYS use when user asks for:** computer-use, screen control, UI automation, app automation, clicking elements, typing text, taking screenshots, or controlling macOS apps programmatically. **Assumption:** Any request involving GUI interaction, screenshots, or app control on macOS should use cua-driver tools via this skill. **DO NOT use for:** headless/scripted automation that doesn't need visual/GUI interaction (use shell scripts instead).

SKILL.md

5.3 KB, as published. Nobody here has run it

cua-computer-use

Computer-use automation via the CuaDriver macOS app. Provides screen control, app automation, and UI interaction capabilities for Pi.

Prerequisites

CuaDriver.app installed at /Applications/CuaDriver.app (v0.1.5+)
macOS Accessibility and Screen Recording permissions granted to CuaDriver

⚡ Quick Start

All tools are invoked via the cua-pi wrapper (auto-starts daemon if needed):

# List all available tools
cua-pi list-apps          # List running/installed macOS apps
cua-pi list-windows       # List all top-level windows with window_ids
cua-pi screenshot '{"window_id":1234}'  # Capture a window screenshot
cua-pi click '{"pid":844,"element_index":5}'  # Click a UI element
cua-pi type-text '{"pid":844,"text":"hello"}'  # Type text into an app
cua-pi press-key '{"pid":844,"key":"return"}'  # Press a key
cua-pi hotkey '{"pid":844,"keys":["command","c"]}'  # Press hotkey combo
cua-pi move-cursor '{"x":100,"y":200}'  # Move mouse cursor
cua-pi get-window-state '{"pid":844,"window_id":1234}'  # Get UI tree as markdown
cua-pi check-permissions   # Verify TCC permission status
cua-pi get-screen-size     # Get display dimensions

Tool names use underscores (e.g. list_apps, get_window_state). The wrapper handles them as-is.

Common Workflows

Inspect an app's UI

Find the app: cua-pi list-apps | jq '.apps[] | select(.name=="Safari")'
Find its window: cua-pi list-windows | jq '.windows[] | select(.app_name=="Safari")'
Get UI tree: cua-pi get-window-state '{"pid":123,"window_id":456}'

Interact with a web page

Get window state to find element indices
Click: cua-pi click '{"pid":123,"element_index":5}'
Type: cua-pi type-text '{"pid":123,"text":"search query"}'
Press key: cua-pi press-key '{"pid":123,"key":"return"}'

Take a screenshot

cua-pi list-windows | jq '.windows[0].window_id'
cua-pi screenshot '{"window_id":<id>,"format":"png"}'

Important Notes

Permissions: If tools silently fail or return empty results, run cua-pi check-permissions. CuaDriver needs both Accessibility and Screen Recording in System Settings → Privacy & Security.
Window ID required: Most interaction tools need a window_id from list-windows.
PID vs Bundle ID: Some tools take pid (process ID), others take bundle_id. Check tool schemas with cua-driver describe <tool>.
Daemon mode: The wrapper auto-starts the cua-driver daemon via open -n -g -a CuaDriver --args serve if not running. The daemon preserves the AppStateEngine cache across calls and ensures TCC permissions are attributed to CuaDriver.app.
JSON arguments: Always pass valid JSON. Use single quotes around the JSON string in bash.

Python Integration

For complex automation, use the Python SDK directly:

# ~/.cua-venv is the virtual environment with cua packages installed
import subprocess, json

def cua_call(tool: str, args: dict) -> dict:
    result = subprocess.run(
        ["/Applications/CuaDriver.app/Contents/MacOS/cua-driver", "call", tool, json.dumps(args)],
        capture_output=True, text=True, check=True
    )
    return json.loads(result.stdout)

# Example: list apps
apps = cua_call("list_apps", {})

Troubleshooting

Symptom	Cause	Fix
`cua-driver: command not found`	PATH issue	Use full path `/Applications/CuaDriver.app/Contents/MacOS/cua-driver`
Empty screenshots	Missing Screen Recording permission	Grant in System Settings → Privacy & Security
Click/type has no effect	Missing Accessibility permission	Grant in System Settings → Privacy & Security
`check_permissions` shows false	TCC not granted	Open CuaDriver.app GUI once, or manually add in System Settings

Common Agent Mistakes

❌ Wrong: Using cua-pi for headless API calls or data extraction ✅ Right: Use curl/httpie for API work; cua-pi is for GUI interaction only

❌ Wrong: Taking screenshots without checking permissions first ✅ Right: Run cua-pi check-permissions before screenshot workflows

❌ Wrong: Using element_index without listing elements first ✅ Right: list-windows → get-window-state → click by index

❌ Wrong: Using cua for tasks better done with shell scripts ✅ Right: Shell scripts for automation; cua only when visual/GUI interaction is needed

Related Skills

@skills/my-web-search-kagi — For web searches when you need to find information, not interact with apps
@skills/my-tech-stack — For tool recommendations (shell scripts vs GUI automation)
@skills/my-workflow — For commit discipline when modifying CuaDriver configs or scripts