Newest papers

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin, Hao Tang, Hung-yi Lee · 2026 · arXiv

DOI: 10.48550/arXiv.2610.28189 · open-access PDF

Chinese

虽然语言模型背骨仅在文本上进行预训练,但大型音频语言模型(LALMs)在很大程度上可以识别语音属性,如情绪,扬声器性别和口语。然而,它不清楚音频表示是如何集成的,如果有,在模型产生文本之前与其他文本表示。我们假设有一个共享的子空间,其中音频和文本表示一致。 为了测试这个假设,我们识别了代表音频和文本的跨模式神经元。

Machine translation (Meta M2M100); the English original follows.

Abstract

While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audi

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange