Newest papers

Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision

Minseong Kweon, Qiao Feng, Yiming Huang, Lingjie Liu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28519 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

了解接触和触摸数据的大小在手对象交互中是向物理智能系统迈出的关键步骤,但学习这种表现受到触摸测量成本和限制的覆盖限制。我们介绍了Touch-JEPA,一个框架来学习触摸图像从手对象视频的相应位置的触摸图像,具有有限的触摸监控。我们首先引入了一种自我监督的预训练客观,它只从视频中学习,而没有触摸监控。使用视频衍生的手地理学,Touch-JEPA预测来自相应的手对象位置的触摸图像特征,从空间和时间背景中鼓励表示捕捉隐形接触动态。然后我们将这个公式扩展到共同表示学习,其中隐形预测应用于所有视频,而有限触摸监控仅适用于一个标签

Machine translation (Meta M2M100); the English original follows.

Abstract

Understanding contact and tactile magnitude in hand-object interaction is a key step toward physically intelligent systems, yet learning such representations is limited by the cost and restricted coverage of tactile measurements. We present Touch-JEPA, a framework for learning tactile representations from hand-object videos with limited tactile supervision. We first introduce a self-supervised pretraining objective that learns from video alone without tactile supervision. Using video-derived hand geometry, Touch-JEPA predicts masked visual features at corresponding hand-surface locations from spatial and temporal context, encouraging the representation to capture latent contact dynamics. We then extend this formulation to joint representation learning, where masked prediction is applied to all videos while tactile supervision is available only for a labeled subset, grounding the shared representation in measured contact and magnitude. With self-supervised pretraining alone, simple supervised readouts on frozen Touch-JEPA features outperform supervised baselines in contact discrimination and tactile magnitude estimation across three human-hand benchmarks, and transfer to robotic hands without robot tactile training data. Joint learning with limited tactile supervision further improves contact and magnitude estimation on the human-hand benchmarks. Together, these results show that hand-object video can provide scalable self-supervision for tactile representation learning, while limited tactile measurements provide complementary physical grounding.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Artificial intelligence, Representation (politics), Computer science, Computer vision, Tactile sensor, Robot

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange