Newest papers

UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception

Linfei Li, Ruining Hu, Lin Zhang, Zhong Wang, Fengyi Zhang, Ying Shen, Binqiang Wang, Xin Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29768 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

La percepción del lenguaje de visión de datos ha logrado un progreso impresionante en alinear el lenguaje natural con las observaciones de marcos visuales, pero fundamentar la semántica de alto nivel en la interacción física de nivel parcial sigue siendo un reto. Para abordar esta brecha, proponemos UniRAP, un modelo unificado para inferir affordances físicas de nivel parcial y mapear instrucciones de lenguaje de contacto ejecutables a representaciones geométricas actuables. UniRAP formula este problema como una tarea de generación condicional multimodal, integrando entradas visuales, instrucciones textuales y prompts opcionales en un espacio de representación interactiva espacial compartida a través de un mecanismo de token de interfaz unificada. Para predecir la geometría de contacto ejecutable, introducimos un Unified Af

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language perception has achieved impressive progress in aligning natural language with visual observations, yet grounding high-level semantics into part-level physical interaction remains challenging. To address this gap, we propose UniRAP, a unified model for inferring part-level physical affordances and mapping language instructions to actionable geometric representations. UniRAP formulates this problem as a conditional multimodal generation task, integrating visual inputs, textual instructions, and optional prompts into a shared spatiotemporal representation space through a unified interface token mechanism. To predict executable contact geometry, we introduce a Unified Affordance Decoder (UAD), which jointly performs object detection, part-level affordance segmentation, and 4-DoF interaction pose estimation by leveraging intermediate segmentation features. In addition, we propose a curriculum-based transfer training strategy that progressively adapts the model from general visual parsing to interaction-aware perception, improving data efficiency under limited textual annotations. Experiments show that UniRAP achieves state-of-the-art performance on referring expression segmentation, affordance grounding, and interaction pose estimation, while maintaining strong spatiotemporal consistency in dynamic video scenarios. These results demonstrate the effectiveness of UniRAP as a unified perception framework for language-guided physical manipulation.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Affordance, Computer science, Executable, Human–computer interaction, Artificial intelligence, Semantics (computer science)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange