Newest papers

From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

Xiaoxu Xu, Hao Li, Hongtao Zhang, Yili Tang, Jiafei Cao, Junhao Cai, Haoxiang Ma, Weixin Li, Weinan Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28164 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

有效的转移需要一个政策,不仅要捕捉当前场景的几何学,而且要了解这种几何学如何在交互中演变。现有方法要么增加政策与当前观察的几何信息,要么引入辅助监督的未来预测,留下开放的预测几何学应该如何代表动作的产生。因此,我们提出了一个两个阶段的框架,该框架然后应用于从多视图视频片段中学到补充几何表现的几何学。在预练过程中,一个冷冻的3D基础模型提供了密集的几何学特征,这些特征被压缩成预测几何学的表现,并优化以保留场景结构和它的时间演变。行动监督任务然后应用于几何学的绝对化,以提取一个补充的引擎代表,强调行动

Machine translation (Meta M2M100); the English original follows.

Abstract

Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction. Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation. Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips. During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution. Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions. During downstream post-training, predictive geometry is transferred through geometry queries, while the motor representation guides intermediate features along the action-generation pathway. Future observations are used only as privileged supervision during training and are not required at inference time. Across LIBERO, LIBERO-Plus, SimplerEnv, and real-world bimanual and single-arm tasks, our approach consistently improves downstream policies, including an absolute gain of 10.4 percentage points on the WidowX evaluation. We further demonstrate transfer across multiple VLA architectures and WAMs, with improvements in both future-world and action prediction. The code will be available at https://github.com/xuxiaoxxxx/CPR.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Representation (politics), Artificial intelligence, Computer science, Action (physics), Inference, Embodied cognition

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange