Newest papers

EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jin’er Wang, Mingze Sun, Ziang Cao, Yalun Dai, Thang Luu, Qin Ma, Hongyi Chen, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26625 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

重建单一摄像机前置对象的强大分类对象从自私的手视频需要更多的,而不是恢复单个3D框架组件:观察者的运动,围绕的几何和手对象的相互作用必须合并在一个一致的世界中。实现这种一致性需要共同重建和调整监督,但现有的方法和数据集提供有限的支持。我们介绍了EgoXperience,一个通过数据收集,自动注释,统一的模型和基准来解决这些需求的全套范式,自动注释和基准。中央到EgoXperience是一个自动注释引擎,EgoX-Engine,它在地理上验证对象跟踪和基于SLAM的世界排列,为摄像头,场景,手,和对象提供一致的公共监督。我们通过

Machine translation (Meta M2M100); the English original follows.

Abstract

Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Artificial intelligence, Computer science, Object (grammar), Computer vision, Task (project management), Coherence (philosophical gambling strategy)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange