Newest papers

Beyond Omni: A Unified MLLM for Embodied Sensory Understanding

Shiyun Xiong, Dongming Wu, Peiwen Sun, Bokang Yang, Jiaxu Wang, Xiaolou Sun, Mingxing Wen, Xiangyu Yue · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26751 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Potensi baru-baru ini sistem embodied semakin membangun pada model bahasa besar multimodal (MLLM) untuk pemahaman dan kemampuan argumen pra-latih mereka, tetapi persepsi mereka tentang dunia fisik tetap sebagian besar terbatas pada modalitas dasar seperti penglihatan dan audio. Namun, memperluas MLLM ke modalitas sensorik yang lebih beragam dari lingkungan fisik adalah tantangan: representasi mereka sering kurang terhubung ke MLLM dan bervariasi secara substansial dalam struktur dan statistik, sementara interpretasi mereka masih memerlukan pemahaman dan alasan tingkat tinggi bersama. Oleh karena itu, kami mengusulkan Unisense, kerangka kerja MLLM yang terintegrasi yang terintegrasi yang menyokong modalitas sensorik yang beragam dalam arsitektur multi-modal bersama. Unisense mengikuti proses pelatihan dua tahap: (i) menyesuaikan representasi sensorik dengan tugas gerakan untuk

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent embodied systems increasingly build on multimodal large language models (MLLMs) for their pretrained understanding and reasoning capabilities, but their perception of the physical world remains largely limited to basic modalities such as vision and audio. However, broadening MLLMs to more diverse sensory modalities from the physical environment is challenging: their representations are often weakly connected to the MLLM and vary substantially in structure and statistics, while their interpretation still requires shared high-level understanding and reasoning. We therefore propose Unisense, a unified embodied MLLM framework that supports diverse physical sensory modalities within a shared multimodal architecture. Unisense follows a two-stage training process: (i) aligning sensory representations with language to establish a language-accessible interface; and (ii) tuning modality-specific experts within the shared LLM to provide specialized processing for different sensory modalities. To support broad sensory learning, we curate Unisense-1M, a large-scale embodied sensory training corpus comprising 1M samples across 8 sensory modalities. We further evaluate Unisense on 12 benchmarks spanning depth, point cloud, IMU, tactile, thermal, radar, motion, and smell understanding. Across these heterogeneous sensing tasks, Unisense achieves strong performance and outperforms both generalist and specialized models on most benchmarks, demonstrating the potential of pretrained MLLMs as unified models for embodied sensory understanding.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Embodied cognition, Sensory system, Perception, Stimulus modality, Embodied agent, Modalities

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange