Shaojin Ma, Ruitao Wang, Pengyi Li, Jie Yang, Liangyu Chen, Gangyi Zhao, Hongyao Tang, Yan Zheng, Jianye Hao · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29427 · cited by 0 works (OpenAlex) · open-access PDF
Die jüngsten Fortschritte im multi-turn-verstärkten Lernen (RL) haben die Leistung von LLM-Agenten bei komplexen, langfristigen interaktiven Aufgaben erheblich verbessert. Trotz des Fortschritts bei der effizienten Zuweisung von Feinkörnern wendet die Richtlinienoptimierung diese Lernsignale immer noch durch antwortspezifische Wahrscheinlichkeitsverhältnisse an. Verschiedene Textreaktionen können den gleichen ausführbaren Effekt hervorrufen, aber die Kreditunterschiede, die sich aus den nachfolgenden Ergebnissen ergeben, bleiben mit ihren individuellen textlichen Realisierungen verbunden. Wir identifizieren diese Mismatch als Kredit-Update-Mismatch. Unsere Analyse zeigt, dass Kredit-Text-Pairing direkt Richtlinien für die
Machine translation (Meta M2M100); the English original follows.
Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks. However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios. Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations. We identify this mismatch as credit-update misalignment. Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed. Motivated by this finding, we propose Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations. EMPO preserves the original credit signals, integrates with both trajectory-level and step-level credit estimators, and requires no additional critics or rollouts. Experiments on ALFWorld, WebShop, and Sokoban show that EMPO outperforms strong baselines in both task performance and learning efficiency.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Reinforcement learning, Computer science, Executable, Context (archaeology), Task (project management), Artificial intelligence
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29427, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange