Newest papers

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning

Nghi Dao, Calder Morton-Ferguson, Noah Golowich, Adam Block · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28770 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

RLVR menetapkan kinerja kredit melalui pertimbangan model bahasa, tetapi umumnya mengandalkan relouts stochastic independen untuk eksplorasi, yang dapat berulang kali sampel efektif strategi yang gagal. Kerja sebelumnya telah melihat kesuksesan dengan pendekatan yang membimbing eksplorasi dengan memperkenalkan ketergantungan ke dalam proses pengambilan sampel dengan memasukkan solusi yang salah dalam konteks untuk upaya akhir. Namun, metode ini memperkenalkan pertanyaan tambahan tentang bagaimana untuk menetapkan kredit untuk setiap upaya. Kami mengusulkan untuk melihat pengambilan kredit melalui lensa Stochastic Shortest Paths, sebuah kerangka kerja dalam pembelajaran penguatan klasik yang bertujuan untuk meminimalkan biaya kumulatif yang diharapkan untuk mencapai kondisi terminal (dalam kasus kami, respon yang terverifikasi). Kami menunjukkan bahwa, di bawah asumsi yang tepat yang menyederhanakan, perspektif ini secara alami mengarah ke modifikasi

Machine translation (Meta M2M100); the English original follows.

Abstract

RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of Stochastic Shortest Paths, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call Retry-Aware Policy Optimization (ReAPO). ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 and OLMo-2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. On Qwen3-0.6B, ReAPO also outperforms GRPO with twice the group size or twice the training steps, reaching their peak accuracy with 39% and 31% fewer generated tokens, respectively. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Context (archaeology), Reinforcement learning, Baseline (sea), Perspective (graphical), Process (computing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange