Newest papers

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Tom Yin, Gabriel Herbert Sarch, Zhuang Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31494 · cited by 0 works (OpenAlex) · open-access PDF

Hindi

दृष्टि-भाषा ड्राइव मॉडल (VLMs) विभिन्न सेटिंग्स पर कार्यों की एक विस्तृत श्रृंखला को पूरा कर सकते हैं, फिर भी एजेंटिक संदर्भों में उनके प्रदर्शन को कम समझा जाता है. हालांकि सिमुलेटर और वास्तविक दुनिया के वातावरण में ऑनलाइन मूल्यांकन इंटरैक्टिव एजेंट प्रदर्शन को मापने के लिए आवश्यक हैं, वे महंगा और स्केल करने के लिए चुनौतीपूर्ण हैं। पूरक ऑफ़लाइन VQA बेंचमार्क एजेंटों, रोबोटिक्स, अग्निसेंटरिक वीडियो, गेम, स्पेस प्लानिंग के पांच डोमेनों पर तेजी से और अधिक विश्वसनीय तुलना की अनुमति दे सकते हैं, फिर भी मौजूदा बेंचमार्क सीमित और विभाजित रहते हैं। इस अंतर

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Computer science, Task (project management), Machine learning, Artificial intelligence, Range (aeronautics)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange