Newest papers

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31544 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Penambahan kontrafaktual memperkirakan kredit ini dengan mengeksekusi kembali episode setelah keputusan alternatif, tetapi cabang menarik bunyi sampel dasarnya independen dari episode faktanya, sehingga perbedaan hasil mencampur efek keputusan dengan keberuntungan dari segala sesuatu yang berikut. Kami memperhatikan bahwa dalam pencarian agentik penambang, corpus, dan grader adalah deterministik, sehingga satu episode adalah fungsi deterministik dari keputusan kebijakan dan bunyi sampelnya. Dunia kembar oleh karena itu dapat diimplementasikan: bunyi yang sama, keputusan yang berbeda. Kami memperkenalkan TWICE (TWIn-world Credit Estimation), yang mengetik bunyi Gumbel dari setiap token dan turn-line, memutar rata-rata poin kontrafaktual yang mewarisi bunyi kontrafaktual dari episode faktanya, dan mengubah perbedaan hasilnya yang disesuaikan menjadi tujuh perbedaan yang diperlukan

Machine translation (Meta M2M100); the English original follows.

Abstract

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-ρ$, where $ρ$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $Δ$ falls from $Θ(Δ^{-2})$ to $Θ(Δ^{-1})$. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Counterfactual thinking, Outcome (game theory), Reinforcement learning, Luck, Computer science, Economics

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange