Newest papers

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, R. An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Ma, Kevin Qinghong Lin, Mike Zheng Shou · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30108 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

多式代理人(MMAs)通过推理、记忆、工具和反馈保持互动,最明显是作为计算机使用的代理人,而机器人系统(RSs)在物理动态下实现任务和行动。我们定义了多式体现代理人(MMEAs)作为目标导向的系统,这些系统通过推理、工具和反馈来对接多式任务,并从结果的反馈中修改决策。这引发了一个核心问题:当多式代理人从计算机使用到机器人使用时发生什么变化?我们引入PAPAV,一个五个重复功能的多式体现型代理人(MMEAs)的统一功能中心框架:感知当前状态,预测行动效果,计划可行的课程,通过接口或任务体,并验证结果。根据功能而不是常见的固定架构

Machine translation (Meta M2M100); the English original follows.

Abstract

Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use? We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome. Defined by function rather than architecture, these capabilities let PAPAV identify what all three share and then compare MMEAs with MMAs and with RSs. Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors. Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success. These constraints also frame the open challenges we identify. PAPAV therefore provides a common basis for designing reliable physical agents and measuring progress across all five capabilities rather than task success alone.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Embodied cognition, Task (project management), Perspective (graphical), Action (physics), Computer science, Observability

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange