Newest papers

SenseAlign: Understanding Vision and Touch through Language at Test Time

Chuyang Ye, Qinting Jiang, Jingyan Jiang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28511 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

在现实世界的部署中,照明、接触质量或传感器硬件的变化可以改变视觉或触摸输入,并降低识别性能。当两种模式有助于预测时,一个不可靠的模式可以通过共享语言空间来识别对象和材料。由于标记数据在部署期间通常是不可用的,测试时间调整提供了从未标记的观察中恢复识别性能的方法。在这种环境中,我们发现,将积累的观察与候选类的文本嵌入物进行比较,可以揭示信任可能错过的模式可靠性差异。针对竞争类别的特征的定义也提高了相同的地理定位之外的识别。这些发现激励了SenligneAn,该研究人员使用一种常见的语义参照来测试时重量和调节功能。我们

Machine translation (Meta M2M100); the English original follows.

Abstract

Tactile-visual-language models combine vision and touch through a shared language space to recognize objects and materials. In real-world deployment, changes in lighting, contact quality or sensor hardware can shift visual or tactile inputs and degrade recognition performance. When both modalities contribute to a prediction, an unreliable modality can override correct evidence from the other. Since labeled data are generally unavailable during deployment, test-time adaptation offers a way to recover recognition performance from unlabeled observations. In this setting, we find that comparing accumulated observations with the text embeddings of candidate classes reveals modality reliability differences that confidence can miss. Refining features against competing classes also improves recognition beyond geometric alignment alone. These findings motivate SenseAlign, which uses a common semantic reference for test-time modality weighting and feature correction. Alignment Trust weights each modality according to its support for shared predicted classes. Anchor Realignment uses the same class text embeddings to fit a shared feature correction that separates predicted classes from alternatives. Both modules use the same unlabeled history while keeping all encoders frozen. Extensive experiments across diverse datasets demonstrate strong recognition performance under both natural shifts and controlled corruptions.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Modality (human–computer interaction), Computer science, Artificial intelligence, Feature (linguistics), Weighting, Modalities

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange