Newest papers

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai‐Wei Chang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26860 · cited by 0 works (OpenAlex) · open-access PDF

Korean

멀티모델 대형 언어 모델(MLLMs)은 시각적 추론에 빠른 진전을 보였지만 많은 작업은 내부 추론만을 통해 얻기 어려운 증거를 필요로 한다: 22개의 벤치마크에 대한 우리의 통제된 연구는 기본 모델의 추론 예산을 단순히 증가시키는 것이 단순한 비용 이득을 가져온다는 것을 보여줍니다. 이러한 작업은 지상식 인식, 외부 지식 및 수치 측정을 필요로합니다. 내부 추론을 통해 얻을 수 있는 증거는 모델에 최대한 해석할 수 있어야 하며 개방적 인 추론 단계를 통해 유지되어야 합니다. 우리는 OpenVLHarness, 기능 계층과 지속적인 멀티모델 메모리 주위에 구성된 멀티모델 추론을 제공합니다. 이러한

Machine translation (Meta M2M100); the English original follows.

Abstract

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Coding (social sciences), Inference, Visual reasoning, Multimodality, Process (computing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange