Newest papers

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28632 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

它基于视频生成模型现在可以描绘令人难以置信的现实运动。然而,它不清楚如何使用这种运动超越像素的地方:动画一个特定的3D资产,并在其3D场景中重新播放动画。例如,一个视频模型可以显示一只猴子跳过十字路口,但它不能返回跳跃作为可重复的3D运动放置在3D场景中。在这里,我们呈现了一个模型,直接共同生成视频和3D运动为任何提供的架构,以及一个运动摄像头,该运动记录这一动作到场景。先前的视频运动模型可以仅处理人类骨骼,经常添加数十亿个参数,或从生成的2D姿势地图中三角形3D运动。我们代替将3D运动保持在一个独立的

Machine translation (Meta M2M100); the English original follows.

Abstract

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Computer vision, Artificial intelligence, Motion (physics), Animation, Computer graphics (images)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange