Gui agent sft qwen2vl os kairos

Skill kjuhwa/skills-hub/skills/llm-agents/gui-agent-sft-qwen2vl-os-kairos

SFT Qwen2-VL-7B on the OS-Kairos GUI dataset using LLaMA-Factory, converting data to ShareGPT format, and running inference.From its SKILL.md

Install

npx -y skills add kjuhwa/skills-hub --skill gui-agent-sft-qwen2vl-os-kairos

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.1 KB, 485 tokens by cl100k_base, as published. Nobody here has run it

GUI Agent SFT: Qwen2-VL-7B on OS-Kairos

When to use

Build a GUI agent that predicts click/type/scroll actions from screenshots.
SFT-training Qwen2-VL-7B on OS-Kairos dataset via LLaMA-Factory.
Model also scores its own action confidence (1-5).

Steps

Download OS-Kairos dataset from https://github.com/Wuzheng02/OS-Kairos
Download Qwen2-VL-7B-Instruct from HuggingFace.
Clone LLaMA-Factory from https://github.com/hiyouga/LLaMA-Factory/
Convert OS-Kairos to ShareGPT format via get_sharpgpt.py (see chapter9/README.md for full prompt template).
Register dataset in LLaMA-Factory data/dataset_info.json:

"Karios": {
  "file_name": "Karios_qwenscore.json",
  "formatting": "sharegpt",
  "columns": {"messages": "messages", "images": "images"},
  "tags": {"role_tag": "role", "content_tag": "content", "user_tag": "user", "assistant_tag": "assistant"}
}

Launch training (requires 3x A100-80GB):

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 FORCE_TORCHRUN=1 \
  llamafactory-cli train examples/train_full/qwen2vl_full_sft.yaml

Run inference:

CUDA_VISIBLE_DEVICES=0 FORCE_TORCHRUN=1 \
  llamafactory-cli webchat examples/inference/qwen2_vl.yaml

Example output

action: CLICK <point>[[454,87]]</point>
score: 5

A lower score signals the task is beyond agent capability and human intervention is needed.

Pitfalls

Training requires at least 3x 80GB A100 GPUs.
model_name_or_path in YAML files must point to local model/checkpoint paths.

Source

Chapter 9 of dive-into-llms - documents/chapter9/README.md

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.