Newest papers

UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

Linfei Li, Ruining Hu, Lin Zhang, Zhong Wang, Fengyi Zhang, Ying Shen, Binqiang Wang, Xin Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29768 · cited by 0 works (OpenAlex) · open-access PDF

French

La perception du langage de vision a réalisé des progrès impressionnants dans l'alignement du langage naturel avec les observations visuelles, mais la fondation de la sémantique de haut niveau dans l'interaction physique au niveau partiel reste un défi. Pour résoudre ce fossé, nous proposons UniRAP, un modèle unifié pour déduire les affordances physiques au niveau partiel et cartographier les instructions du langage de contact à des représentations géométriques actionables. UniRAP formule ce problème en tant que tâche de génération multimodale conditionnelle, intégrant les entrées visuelles, les instructions textuelles et les prompts optionnels dans un espace de représentation interactive spatiotemporelle partagé à travers un mécanisme de token d'interface unifi

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Affordance, Computer science, Executable, Human–computer interaction, Artificial intelligence, Semantics (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange