Newest papers

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning

Nghi Dao, Calder Morton-Ferguson, Noah Golowich, Adam Block · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28770 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

RLVR asigna el desempeño de crédito a través de el razonamiento del modelo de lenguaje, pero generalmente se basa en los lanzamientos estocásticos independientes para la exploración, que pueden repetidamente asignar muestras eficaces de estrategias fallidas similares. El trabajo anterior ha visto el éxito con enfoques que guían la exploración introduciendo dependencia en el proceso de muestreo mediante la inclusión de soluciones incorrectas en el contexto de los intentos posteriores. Sin embargo, estos métodos introducen la pregunta adicional de cómo asignar crédito a cada intento. Proponemos ver la asignación de crédito a través de la lente de los caminos más cortos estocásticos, un marco en el aprendizaje de reforzamiento clásico que busca minimizar el coste acumulativo esperado para llegar a un estado terminal (en nuestro caso

Machine translation (Meta M2M100); the English original follows.

Abstract

RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of Stochastic Shortest Paths, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call Retry-Aware Policy Optimization (ReAPO). ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 and OLMo-2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. On Qwen3-0.6B, ReAPO also outperforms GRPO with twice the group size or twice the training steps, reaching their peak accuracy with 39% and 31% fewer generated tokens, respectively. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Reinforcement learning, Baseline (sea), Perspective (graphical), Process (computing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange