Newest papers

Inside-JEPA: From Prediction to Relational Grounding in Latent World Models

Hossein Resani, Javen Qinfeng Shi · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31194 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Las arquitecturas predictivas de incorporación correcta conjunta (JEPAs) aprenden modelos del mundo en el espacio latente, permitiendo una planificación eficiente pero dejando sus estados opacos. Introducimos Inside-later, que coloca canales de relación para pares de entidades ordenadas dentro del estado latente de JEPA y muestra que una pérdida de dinámica inversa puede aterrizar estos canales sin etiquetas de relación cuando su cabeza los lee, solo o como parte del estado completo. En Something-Something V2, la predicción latente sola no funciona mejor que una cabeza de relación aleatoria, mientras que la supervisión de la dinámica inversa que recupera acciones de los cambios de canal mejora el AUROC relativo en .065-.103 casos en diez configuraciones con cajas anotadas, por .0 en los episodi

Machine translation (Meta M2M100); the English original follows.

Abstract

Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Relation (database), Artificial intelligence, Action (physics), Relational database, Computer science, Statistical relational learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange