Newest papers

Large Multimodal Models for Direct Robot Control: GPT-6 as a Beginning

Youwei Zhou, Bohan Hou, Yankai Fu, Yaoxu Lyu, Yifan Ye, Shilin Shan, Chen Guo, Jiaming Liu, Ran He, Caifeng Shan, Shanghang Zhang, Jianfei Yang, Zhihe Lu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28259 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

LIBworldは、視野空間の理解と代理理論の最近の進歩を妨げているが、一般的な目的の多様なモデルで直接的なロボット操作を可能にしますが、その実践的な能力は未だに理解されていません。この論文では、GPT-6 Astraは、閉鎖ロボット操作のための閉鎖ロボット操作のためのシステム的に評価され、軽量エージェントの実行枠組みを使用して反応を使用します。私たちは、最初に3つの基本的な実体化された能力 - 深さ感知、コンテキストの学習、およびタクティルタクティルタクティルの理解を体現し、このモデルは深さ情報を活用し、文脈的なデモから単純な操作戦略を取得し、その行動を修正するためにタクティルフィ

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent advances in visuospatial understanding and agentic reasoning enable direct robotic manipulation with general-purpose multimodal models, but their embodied capabilities remain poorly understood. This paper systematically evaluate GPT-6 Astra for closed-loop robotic manipulation using a lightweight agent execution framework. We first probe three fundamental embodied capabilities-depth perception, embodied in-context learning, and tactile understanding-and demonstrate that the model can exploit depth information, acquire simple manipulation strategies from contextual demonstrations, and use tactile feedback to correct its actions. We then evaluate GPT-6 Astra across five established manipulation benchmarks along six capability dimensions. Despite competitive performance on several benchmarks-26.92\% success on RoboDojo and 76.0\% on RoboTwin 2.0-our analysis reveals a pronounced gap between semantic reasoning and physical execution, particularly in consequence-aware action planning, precision, and complex motion coordination. To further examine these limitations, we introduce LIBERO-GeoMotion, a new benchmark of geometry-constrained tasks that require precise pose alignment, coordinated translation and rotation, and anticipation of physical action consequences. GPT-6 Astra achieves only 11.1\% zero-shot success, exposing the difficulty of translating spatial understanding into physically feasible motion. Real-world experiments corroborate these findings: zero-shot execution is feasible for simple tasks, yet persistent deficits remain in motion feasibility, precise control, and physical safety, while substantial inference latency further hinders practical deployment. Overall, our results characterize the emerging embodied capabilities of general-purpose multimodal models and delineate the key challenges toward safe, efficient, and reliable robotic manipulation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Embodied cognition, Computer science, Exploit, Human–computer interaction, Artificial intelligence, Action (physics)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange