Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou · 2026 · arXiv
DOI: 10.48550/arXiv.2610.29999 · open-access PDF
Pembelajaran penguatan (RL) berdasarkan pemotongan gaya PPO telah menjadi pendekatan yang efektif untuk meningkatkan kemampuan pemikiran dari model bahasa besar (LLM). Namun, langkah-langkah pemotongan standar tingkat token mengubah kebijakan hanya melalui probabilitas kondisional token saat ini, sementara konteks kondisional prefix itu sendiri dihasilkan oleh kebijakan. Ketidakseimbangan ini menjadi terutama konsekuensional dalam pelatihan respon panjang dan off-policy. Kami mereformulasikan generasi autoregresif sebagai proses keputusan Markov yang terbatas-horizon dan menghasilkan identitas prestasinya-diferensi di bawah perilaku-poli
Machine translation (Meta M2M100); the English original follows.
Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-poli
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29999, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange