Newest papers

Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

Noam Rotstein, Yoad Tewel, Chen Tessler, Haggai Maron, Gal Chechik, Or Litany, Yoni Kasten, Yuval Atzmon · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28632 · cited by 0 works (OpenAlex) · open-access PDF

Korean

그것은 동영상 생성에 기반한 유효한 컨트롤 모델을 이제 상당히 현실적인 움직임을 묘사 할 수 있습니다. 그럼에도 불구하고 픽셀 외에이 움직임을 사용하는 방법은 명확하지 않습니다 : 특정 3D 자산을 동기화하고 3D 장면에서 동작을 재생합니다. 예를 들어, 동영상 모델은 동영상 장면에이 움직임을 기록하는 움직임 카메라와 함께 동작을 재생할 수 있습니다. 사전 동영상 모션 모델은 인간의 척추 만 처리 할 수 있지만, 종종 수십억 개의 매개 변수를 추가하거나 생성 된 2D 포즈 맵에서 재생할 수있는 3D 움직임을 삼각화 할 수 없습니다. 여기 우리는 자발적 인 스케레셋을 지원하는

Machine translation (Meta M2M100); the English original follows.

Abstract

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Computer vision, Artificial intelligence, Motion (physics), Animation, Computer graphics (images)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange