Newest papers

Learning the 4D World through Point Trajectory Forecasting

Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28594 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

可靠的多观点预测需要来自不同摄像头的视频来描述同一正在演变的3D场景。 为此,我们引入了视频点模型(VPM),一个世界模型,该模型要求多观点视频历史和语言指示,共同预测来自任意观点的未来的视频,以及持续场景点的3D轨道。我们扩展了一个预训练的世界模型,使用相机调节的多观点处理,该模型集成了相机间的补充观察,并添加了一个轨道头,预测物理点如何通过时间移动。每条轨道都在所有观点上追踪一个单一的点,并监督这些轨道,因此在每个摄像头中将外观变化引入3D运动的共同描述。为了支持训练和运动评估,我们构

Machine translation (Meta M2M100); the English original follows.

Abstract

Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene. To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points. We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time. Each trajectory tracks a single point across all views, and supervising these trajectories alongside video generation therefore anchors appearance changes in every camera to a shared description of 3D motion. To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments. We further extend VPM to robot control by adding robot actions as a token modality in the generator. Action tokens share the point-supervised backbone and are denoised jointly with future videos, thereby grounding the policy in the same 3D-aware representation. On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Trajectory, Artificial intelligence, Computer vision, Viewpoints, Benchmark (surveying)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange