Newest papers

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31544 · cited by 0 works (OpenAlex) · open-access PDF

French

Le déploiement contre factuel estime ce crédit en réexécutant l'épisode après une décision alternative, mais la branche tire son bruit d'échantillonnage indépendamment de l'épisode factuel, de sorte que la différence de résultat mélange l'effet de la décision avec la chance de tout ce qui suit. Nous observons que dans la recherche contre factuel le retriever, le corpus et le gradient sont déterministes, de sorte qu'un épisode est une fonction déterministe des décisions de la politique et de son bruit d'échantillonnage. Les mondes jumelles sont donc réalisables: le même bruit, une décision différente. Nous introduisons TWICE (TWIn-world Credit Estimation), qui clôture le bruit

Machine translation (Meta M2M100); the English original follows.

Abstract

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-ρ$, where $ρ$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $Δ$ falls from $Θ(Δ^{-2})$ to $Θ(Δ^{-1})$. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Counterfactual thinking, Outcome (game theory), Reinforcement learning, Luck, Computer science, Economics

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange