Newest papers

Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning

Zebin Xing, Yuhang Zheng, Yupeng Zheng, Yichen Zhang, Zichen Li, Shangqing Zhou, Yujie Zang, Yuchen Liu, Zhongpu Xia, Haoran Li, Qichao Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28613 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Evaluar y post-entrenar las políticas de robots a través de la interacción física es costoso y difícil de escalar, motivando los modelos del mundo condicionados por la acción como simuladores interactivos. Sin embargo, las predicciones visualmente plausibles de la dinámica del contacto pueden falsear, proporcionando un feedback poco fiable para la evaluación y el aprendizaje de las políticas. Introducimos PhysCanvas, un modelo del mundo visuo-táctil coherente con el estado para la manipulación de robots ricos en contacto. Estas imágenes de movimiento de PhysCanvas unifican entre las imágenes multi-visual RGB, las imágenes táctiles basadas en el valor, el desplazamiento del marcador, el estado de la tarea y las acciones dentro de un espacio de visión unificado. Todas las modalidades comparten un vídeo VAE y una

Machine translation (Meta M2M100); the English original follows.

Abstract

Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases $π_{0.5}$'s mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning. Project page: https://zebinx.github.io/PhysCanvas_HomePage/.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Artificial intelligence, Modality (human–computer interaction), Task (project management), Robot, Robotics

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange