Newest papers

SenseAlign: Understanding Vision and Touch through Language at Test Time

Chuyang Ye, Qinting Jiang, Jingyan Jiang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28511 · cited by 0 works (OpenAlex) · open-access PDF

Korean

실제 배포에서 조명, 접촉 품질 또는 센서 하드웨어의 변화는 시각 또는 터치 입력을 변경하고 인식 성능을 악화시킬 수 있습니다. 두 가지 모드가 예측에 기여할 때, 신뢰할 수없는 모드가 서로에게서 올바른 증거를 초과 할 수 있습니다. 라벨 된 데이터가 배포하는 동안 일반적으로 사용할 수 없기 때문에, 테스트 타임 적응은 비 라벨 된 관찰에서 인식 성능을 회복하는 방법을 제공합니다. 이 환경에서, 우리는 두 가지 모드가 예측에 기여하면 신뢰할 수없는 모드의 신뢰성 차이를 밝힐 수 있습니다. 라벨 된 데이터는 배포하는 동안 일반적으로 사용할 수 없기 때문에, 테스트 타임 적응은 동일한 기하학적 조정 외의 인식을 향상

Machine translation (Meta M2M100); the English original follows.

Abstract

Tactile-visual-language models combine vision and touch through a shared language space to recognize objects and materials. In real-world deployment, changes in lighting, contact quality or sensor hardware can shift visual or tactile inputs and degrade recognition performance. When both modalities contribute to a prediction, an unreliable modality can override correct evidence from the other. Since labeled data are generally unavailable during deployment, test-time adaptation offers a way to recover recognition performance from unlabeled observations. In this setting, we find that comparing accumulated observations with the text embeddings of candidate classes reveals modality reliability differences that confidence can miss. Refining features against competing classes also improves recognition beyond geometric alignment alone. These findings motivate SenseAlign, which uses a common semantic reference for test-time modality weighting and feature correction. Alignment Trust weights each modality according to its support for shared predicted classes. Anchor Realignment uses the same class text embeddings to fit a shared feature correction that separates predicted classes from alternatives. Both modules use the same unlabeled history while keeping all encoders frozen. Extensive experiments across diverse datasets demonstrate strong recognition performance under both natural shifts and controlled corruptions.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Modality (human–computer interaction), Computer science, Artificial intelligence, Feature (linguistics), Weighting, Modalities

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange