Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.31274 · cited by 0 works (OpenAlex) · open-access PDF
Multimodal Large Language Models는 컴퓨터 사용 작업에서 유망한 성능을 보여주었으며, 사용자 작업 흐름을 계획하여 자연 언어 작업을 토지하기까지 정밀한 UI 목표에 도달합니다. 기존의 end-to-end 벤치마크는 주로 long-horizon 작업 완료와 GUI 토지 벤치마크가 전통적인 제어 및 제한된 전문 문자 인터페이스에 초점을 맞추고 있습니다. 우리는 POINTERBENCH (POINTERBENCH-Text) 문서에서 문자 수준의 토지와 100 개의 공공용 스크린 샷 지침 쌍 (POINTERBENCH-Pro)의 혼합 전문 GUI 토지를 소개합니다. 우리는 표준 표지판에서 얇은 하위 요소 식별 및 지오메트릭 토
Machine translation (Meta M2M100); the English original follows.
Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Ground, Computer science, Suite, Workflow, Task (project management), Benchmark (surveying)
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.31274, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange