Newest papers

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman, Samuel Okafor · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31544 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

反事实分支估计通过在替代决定后重新运行该事件,但分支取出其样本噪音独立于事实事件,所以结果差异混合了决定的效果,一切随之而来的运气。我们观察到,在代理的搜索中,回收者,尸体和分支器是确定性的,所以一个事件是政策决策和其采样噪音的确切决定性功能。因此,双胞胎世界是可实现的:相同的噪音,不同的决定。我们引入了TWICE(TWIn-world Credit Estimation),它将每一个代币的Gumbel噪音按线下键,滚动出反事实的平均值,它继承了事实事件的QWIN下流噪音,并将其改进的结果差异转化为在独立的对比层面对应的

Machine translation (Meta M2M100); the English original follows.

Abstract

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by $1-ρ$, where $ρ$ is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size $Δ$ falls from $Θ(Δ^{-2})$ to $Θ(Δ^{-1})$. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Counterfactual thinking, Outcome (game theory), Reinforcement learning, Luck, Computer science, Economics

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange