agentsclimarketplace

Screen interaction

Skill mustafaakben/plugin-windows-mcp/skills/screen-interaction

This skill should be used when the user asks to "click a button", "type text", "scroll a page", "move the mouse", "press a keyboard shortcut", "interact with GUI elements", "automate mouse clicks", "fill in a text field", "use keyboard shortcuts on Windows", or needs to perform any direct screen interaction using Windows MCP tools.From its SKILL.md

Install
npx -y skills add mustafaakben/plugin-windows-mcp --skill screen-interaction

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

5.4 KB, as published. Nobody here has run it

Screen Interaction with Windows MCP

This skill covers the six tools for direct GUI interaction: Click, Type, Scroll, Move, Shortcut, and Wait. These tools form the foundation of all visual Windows automation.

Golden Rule: Observe Before Acting

Never click, type, or scroll blindly. Always capture the screen state first:

  1. Use Snapshot to get interactive element IDs and coordinates
  2. Identify the target element from the snapshot results
  3. Perform the interaction using the element's coordinates
  4. Verify the result with Screenshot or another Snapshot

Click

Perform mouse clicks at screen coordinates.

Key parameters: x, y (required), button (left/right/middle), double_click (boolean)

Patterns:

  • Single left-click: activate buttons, select items, set focus
  • Double-click: open files, select words in text
  • Right-click: open context menus

Workflow:

Snapshot -> identify element coordinates -> Click(x, y) -> Screenshot to verify

Type

Input text into the currently focused element.

Key parameters: text (required), clear (boolean — clears field before typing)

Patterns:

  • New input: Click on field, then Type(text="value")
  • Replace existing text: Click on field, then Type(text="new value", clear=true)
  • Multi-line input: include newline characters in the text string

Important: Always ensure the target field has focus (via Click) before calling Type. Without focus, text may be typed into the wrong element.

Scroll

Scroll vertically or horizontally within a window or region.

Key parameters: direction (up/down/left/right), amount, x/y (coordinates)

Patterns:

  • Scroll within a specific panel: provide x/y coordinates inside that panel
  • Page through a document: use larger amount values
  • Scroll to reveal hidden content: moderate amount, then verify with Screenshot

Best practice: Specify coordinates to target a particular scrollable area. Without coordinates, scrolling applies to the window under the cursor.

Move

Move the mouse pointer, with optional drag support.

Key parameters: x, y (required), drag (boolean)

Patterns:

  • Hover: Move(x, y) — triggers tooltips, hover menus, highlights
  • Drag and drop: Click at source, then Move(x, y, drag=true) to destination
  • Slider adjustment: Click on slider handle, Move with drag=true

Shortcut

Execute keyboard shortcuts — often more reliable than GUI clicks.

Key parameters: keys (required — e.g., "ctrl+c", "alt+tab")

Essential Windows shortcuts:

ShortcutAction
ctrl+c / ctrl+vCopy / Paste
ctrl+z / ctrl+yUndo / Redo
ctrl+sSave
ctrl+aSelect all
alt+tabSwitch window
alt+f4Close window
win+dShow desktop
win+eOpen File Explorer
ctrl+shift+escTask Manager
win+rRun dialog
ctrl+wClose tab
ctrl+tNew tab (browsers)

When to prefer Shortcut over Click:

  • Menu actions (File > Save -> prefer ctrl+s)
  • Tab/window switching (prefer alt+tab)
  • Text operations (prefer ctrl+c, ctrl+v, ctrl+a)
  • Any action with a well-known keyboard shortcut

Wait

Pause execution to allow the UI to catch up.

Key parameter: duration (seconds)

Recommended wait times:

ScenarioDuration
Between rapid UI actions0.2-0.5s
After clicking a menu item0.3-0.5s
After switching tabs/windows0.5-1s
After launching an application1-3s
After triggering a dialog/popup0.5-1s
After saving a large file1-2s

Prefer verification over long waits. Instead of Wait(5), use Wait(1) + Screenshot to check if the UI has updated, then proceed or wait more.

Common Interaction Patterns

Fill a Form

Snapshot -> identify fields
Click(field1_x, field1_y) -> Type(text="value1")
Click(field2_x, field2_y) -> Type(text="value2")
Click(submit_x, submit_y)
Screenshot -> verify submission

Navigate a Menu

Click(menu_x, menu_y) -> Wait(0.3)
Snapshot -> identify menu items
Click(item_x, item_y)

Copy Text from One App to Another

App(switch to source) -> Wait(0.5)
Shortcut("ctrl+a") -> Shortcut("ctrl+c")
App(switch to target) -> Wait(0.5)
Click(target_field_x, target_field_y)
Shortcut("ctrl+v")

Drag and Drop

Snapshot -> identify source and destination
Click(source_x, source_y)
Move(dest_x, dest_y, drag=true)
Screenshot -> verify result

Error Recovery

  • Clicked wrong element: Use Shortcut("ctrl+z") to undo if possible, then retry
  • Typed in wrong field: Use Shortcut("ctrl+z"), click the correct field, retry
  • Menu disappeared: Click the menu trigger again, use Wait(0.3) before clicking items
  • Dialog appeared unexpectedly: Use Snapshot to identify dialog buttons, dismiss or respond

Additional Resources

Reference Files

For detailed interaction patterns and advanced techniques:

  • references/patterns.md — Common multi-step interaction patterns, edge cases, and troubleshooting

Keep looking

Skills are one crate of 608,475. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.