Newest papers

VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

Yuto Shibata, Kotaro Amaya, Kashu Yamazaki, Lalit Jayanti, Yoshimitsu Aoki, Mariko Isogawa, Katerina Fragkiadaki · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28523 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

交互性接触者与人之间的交互性OCACT需要机器人在严重的部分可观察性下运作:在密切的物理接触中,人类伴侣被隐藏、断裂或在机器人视野之外,导致交互性交互性估计失败,并经常需要外部跟踪或特权的人类状态。同时,重度的相互感官遮蔽会引入用于学习的人与人运动捕捉数据中的错误。我们介绍REACT,一个可部署的人与人交互政策的框架,同时从四个人与人交互的示范例中进行学习。REACT将语义接触结构作为关键的监控信号连接过程,互动学习和板上控制。首先,VLM引导的接触模型和重定向管道识别有意义的接触者,并将其转

Machine translation (Meta M2M100); the English original follows.

Abstract

Humanoid-human interaction requires robots to operate under severe partial observability: during close physical contact, the human partner is occluded, truncated, or outside the robot's field of view, causing pose estimation to fail and often necessitating external tracking or privileged human state. At the same time, heavy mutual occlusions introduce errors into the human-human motion-capture data used for learning. We present REACT, a framework for learning deployable humanoid-human interaction policies from human-human demonstrations. REACT treats semantic contact structure as the key supervisory signal connecting demonstration processing, interaction learning, and onboard control. First, a VLM-guided contact modeling and retargeting pipeline identifies semantically meaningful contacts and transfers them onto humanoid-human geometry while preserving interaction structure. Using these annotations, privileged interaction policies learn physically grounded behaviors through interaction-aware contact rewards and are then distilled into deployable egocentric depth policies trained entirely from onboard sensing using photorealistic human depth rendering. To address severe partial observability, we introduce generative partner-state modeling objectives that predict partner motion from occluded egocentric views. We evaluate REACT in simulation and on a real Unitree G1 humanoid across five contact-rich interactions. Semantic contact modeling substantially improves preservation of demonstrated interactions, while generative partner-state modeling improves contact recall on four of five interactions, most under heavy occlusion. On real hardware, REACT performs all five interactions using onboard sensing alone.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Retargeting, Computer science, Human–computer interaction, Artificial intelligence, Human–robot interaction, Pipeline (software)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange