Newest papers

EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

Xinhao Ji, Hui En Pang, Hao Li, Kairui Hu, Jin’er Wang, Mingze Sun, Ziang Cao, Yalun Dai, Thang Luu, Qin Ma, Hongyi Chen, Yukang Cao, Fangzhou Hong, Zhaoxi Chen, Ziwei Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26625 · cited by 0 works (OpenAlex) · open-access PDF

Korean

실패한 카메라 전면 카메라 활동을 복구하는 것은 개인 3D 프레임 구성 요소를 복구하는 것보다 더 많은 것을 필요로합니다 : 관찰자의 움직임, 주변 기하학 및 손 객체 상호 작용은 일관된 세계에 맞춰야합니다. 이 일관성을 달성하는 것은 공동 재구성과 일관된 감독을 필요로하지만 기존의 방법과 데이터 세트는 각각에 대한 제한된 지원을 제공합니다. 우리는 카메라, 장면, 성능, 자동 메트 코팅, 통일 된 모델링 및 벤치마킹을 통해 이러한 요구를 충족시키는 전체 스택 패러다임인 EgoXperience을 제시합니다. EgoX Experience의 중앙은 자동 인코팅 엔진인 EgoX-Engine이며, 존재하는 방법과 데이터 세트는 SLAM 기반

Machine translation (Meta M2M100); the English original follows.

Abstract

Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Artificial intelligence, Computer science, Object (grammar), Computer vision, Task (project management), Coherence (philosophical gambling strategy)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange