Newest papers

AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30097 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los modelos de VLA han mostrado promesa en la manipulación robótica, pero el aprendizaje de imitación (IL) solo a menudo lucha para dominar las habilidades complejas de largo horizonte. El aprendizaje de rendimiento reforzado (RL) ofrece una vía efectiva para mejorar aún más las políticas de VLA. Sin embargo, los métodos VLA-RL existentes suelen sufrir de la asignación de crédito retrasada y biasada debido a recompensas escasas en el nivel de la trayectoria, mientras que estos dominios de subtascas desequilibrados crean barreras de botella que limitan el rendimiento general. Para abordar estos desafíos, proponemos AtomicRL, un marco de post-entrenamiento RL que aprovecha las estructuras de tareas atómicas para la asignación de crédito fiable y la exploración

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Reinforcement learning, Computer science, Task (project management), Artificial intelligence, Pipeline (software), Imitation

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange