Winguictl

Windows Desktop Automation with winguictl

⚠️ Security Notice

This skill directly controls Windows desktop through mouse/keyboard simulation. Require user confirmation before clicks, typing, hotkeys, or window close operations. Use --dry-run to preview. Close sensitive apps before use.

Script Path

python <SKILL_DIR>\scripts\winguictl.py <command> [options]

Commands

Command	Purpose	Documentation
window	List/focus/minimize/maximize/close windows	window.md
snapshot	Get HWND/UIA/OCR structure	snapshot.md
find	Find elements by text/UIA/OCR/image	find.md
action	Click/drag/type/scroll/hotkey	action.md
control	Win32 control operations	control.md
uia-control	UIA element operations	control.md
screenshot	Capture window screenshots	screenshot.md
wait	Wait for conditions	wait.md
clipboard	Copy files/text, get text	clipboard.md

Parameter Types

Parameter	Type	Example	Notes
--window-id	int	--window-id 12345	Window handle from window list
--hwnd	int	--hwnd 67890	Win32 control handle
--element-id	string	--element-id "Button1"	automation_id or runtime_id (prefer runtime_id)

Core Workflow

# 1. Identify window
python scripts\winguictl.py window list

# 2. Focus window
python scripts\winguictl.py window --window-id <id> focus

# 3. Get snapshot
python scripts\winguictl.py snapshot --window-id <id> uia

# 4. Find & interact
python scripts\winguictl.py find --window-id <id> uia --text "Submit"
python scripts\winguictl.py uia-control --window-id <id> --element-id <elem_id> click

# 5. Verify (re-snapshot)
python scripts\winguictl.py snapshot --window-id <id> uia

Decision Guide

Locator Priority

Priority	Method	Command	Reliability
1	HWND (Win32)	control --hwnd <hwnd> click	Highest
2	runtime_id (UIA)	uia-control --element-id <id> click	High
3	automation_id (UIA)	uia-control --element-id <id> click	Medium
4	Image matching	action click-image --image-path <path>	Medium
5	Coordinates	action click --relative-x/y	Lowest

Click Method Selection

Command	Mechanism	Best For
uia-control click	UIA InvokePattern	Standard Windows controls, WinUI3
action click --element-id	Mouse simulation at center	WeChat, custom controls, SplitButton, ToolBar

Recommendation: Try uia-control click first. If it fails with errors like 'UIAWrapper' object has no attribute 'click' or returns ok: false, immediately fall back to action click --element-id.

Common Scenarios

Scenario	Command
Win32 control with hwnd	control --hwnd <hwnd> click
UIA element with runtime_id	uia-control --element-id "42-3155764" click
Text-based element	find ocr "text" → action click --element-id
Qt applications	Add --skip-actions --skip-state for faster UIA
SplitButton / ToolBar (save dialog toolbar)	action click --element-id (NOT uia-control)
Set text in edit/combo box	uia-control set-text "value" --verify-change

Key Rules

• Re-snapshot after actions: UI state changes after clicks/typing
• Use --dry-run: Preview operations before execution
• Always prefer control/uia-control over action: More reliable for standard Windows controls, WinUI3, and custom controls that implement UIA interfaces.
• Prefer runtime_id: More reliable than automation_id
• Report window_id: Always include exact window identifier in results
• Coordinate system: relative_rect = window-relative; absolute_rect = screen-absolute

Application Guides

• WeChat Automation — WeChat 4.1.6+ messaging, contacts, files, calls, Moments
• Standard File Dialog — Windows "打开/另存为" dialog automation

References

• Dependencies — pywinauto, pywin32, Pillow, wx-ocr (optional)
• Output Format — JSON structure, exit codes, boundary markers
• Coordinates — Coordinate system details
• Security Guidelines — Detailed safety practices