Newest papers

V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos

Ruicheng Li, Qixiu Li, Yu Deng, Sicheng Xu, Jiaolong Yang, Baining Guo · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26780 · cited by 0 works (OpenAlex) · open-access PDF

German

Egocentrisches Kontext-Single-Video unterstützt vielfältige Anwendungen, darunter Roboterlernen, Augmented Reality und das Verständnis der menschlichen Aktivität. Die Wiederherstellung von metrischen 3D-Handbahnen aus einem solchen Video erfordert die Schätzung sowohl der Handbewegung als auch der niedrigeren Kamerahandhabung, um die Hände in einem konsistenten Referenzrahmen auszudrücken. Viele bestehende Pipelines erholen Hand- und Kamerahandhabung in getrennten Phasen, indem sie Handdetektion, Pipperrekonstruktion, Kamera-Tracking und Trajectory-Verfeinerung kombinieren. Wir präsentieren V-JEPA Hand, ein einfaches Feed-Forward-Video-Modell, das die bilaterale Handbewegung und

Machine translation (Meta M2M100); the English original follows.

Abstract

Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding. Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame. Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement. We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips. It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion. We use the predicted camera motion to express the hand trajectories in a common world frame. The model accepts known focal lengths or predicts them when unavailable. It reconstructs each clip in a single forward pass. We train both components end to end in a single stage on a mixture of diverse pseudo-labelled videos and high-precision 3D annotations. A single model achieves the lowest coverage-aware mean per-joint position error (MPJPE) among comparable baselines on ARCTIC, HOT3D, and EgoStandard. It also yields lower world-frame trajectory errors than the compared SLAM-based pipelines. Ablations support the benefits of video pretraining, temporal context, and mixed supervision. V-JEPA Hand achieves a forward-pass throughput of 93.4 FPS on a single RTX A6000. Project page: https://qft-333.github.io/V-JEPA-Hand-Page/.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer vision, Artificial intelligence, Computer science, Trajectory, Motion (physics), Camera auto-calibration

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange