Newest papers

Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang, Xiaolou Sun, Mingxing Wen, Xiangyu Yue · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26751 · cited by 0 works (OpenAlex) · open-access PDF

Korean

최근의 잠재적 인 구체화 된 시스템은 이전에 훈련 된 이해 및 추론 능력을 위해 멀티모델 대 언어 모델 (MLLM)을 점점 더 많이 구축하지만 물리적 세계에 대한 그들의 인식은 아직도 시각 및 오디오와 같은 기본 모듈에 한정되어 있습니다. 그러나, 멀티모델 대 언어 모델 (MLLM)을 물리적 환경에서 더 다양한 감각 모듈로 확장하는 것은 도전적입니다. 그들의 표현은 종종 공유 멀티모델 아키텍처 내에서 다양한 물리적 감각 모듈을 지원합니다. Unisense는 두 단계의 교육 과정을 따릅니다 : (i) 움직임과 감각 표현을 조정하여 언어에 접근 할 수있는 언어 인터페이스를 구축하는 작업을 수행합니다. 그리고 (ii)

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, Unisense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Embodied cognition, Sensory system, Perception, Stimulus modality, Embodied agent, Modalities

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange