Newest papers

Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang, Xiaolou Sun, Mingxing Wen, Xiangyu Yue · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26751 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

最近的潜在体现系统越来越多地基于多式大语言模型(MLLMs)以其预先训练的理解和推理能力,但他们对物理世界的感知仍然大大局限于视觉和音频等基本模式。然而,将MLLM扩展到从物理环境中更多样化的感官模式是具有挑战性的:他们的代表往往与MLLM相连,在结构和统计方面有很大差异,而他们的解释仍然需要共享的高层次的理解和推理。因此,我们建议Unisense,一个统一的体现在的MLLM框架,支持在共享的多式架构中多样化的物理感官模式。Unisense遵循一个两个阶段的培训过程:(i)将感官表示与动作语言相匹配,以建立一个语言可访问的界面;(ii)

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, Unisense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Embodied cognition, Sensory system, Perception, Stimulus modality, Embodied agent, Modalities

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange