Newest papers

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin, Hao Tang, Hung-yi Lee · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28189 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Sementara tulang belakang model bahasa diprediksi hanya pada teks saja, model audio-bahasa besar (LALM) dapat, pada tingkat yang besar, mengenali atribut akustik, seperti emosi, gender pembicara, dan bahasa berbicara. Namun, tidak jelas bagaimana representasi audio terintegrasi, jika sama sekali, dengan representasi teks lainnya sebelum model menghasilkan teks. Kami mengasumsikan bahwa ada subspasi bersama di mana representasi audio dan teks disesuaikan. Untuk menguji hipotesis ini, kami mengidentifikasi neuron lintas-modal yang bertanggung jawab untuk mewakili keduanya audio dan teks. Kami menemukan penyesuaian antara neuron audio dan teks ke tingkat yang berbeda di tiga LALM dan empat jenis atribut akustik. Intervensi penyebab, mengarahkan dan menonaktifkan neuron, mengkonfirmasi bahwa neuron lintas-modal tidak hanya diperlukan untuk prediksi

Machine translation (Meta M2M100); the English original follows.

Abstract

While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Representation (politics), Speech recognition, Subspace topology, Natural language processing, Artificial intelligence

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange