使用 OmniParser 进行基于视觉的 GUI 解析
Parse screenshots into structured UI elements so computer-use agents can reason about controls before acting.
Prerequisites
Python 3.12; conda; Hugging Face model weights; optional Gradio demo
Installation
Use the upstream install or setup path that matches your environment:
- conda create -n "omni" python==3.12
- conda activate omni
- pip install -r requirements.txt
Requirements and caveats from upstream:
- python
- python weights/convert_safetensor_to_pt.py
- python gradio_demo.py
Basic usage or getting-started notes:
- First clone the repo, and then install environment:
- cd OmniParser
- Ensure you have the V2 weights downloaded in weights folder (ensure caption weights folder is called icon_caption_florence). If not download them with:
- Source: https://github.com/1991513ccie-png/skills
- Extracted from upstream docs: https://raw.githubusercontent.com/microsoft/OmniParser/HEAD/README.md
Documentation
- https://microsoft.github.io/OmniParser/
