Newest papers

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29999 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

基于PPO风格的剪辑,Reinforcement Learning(RL)已成为提高大语言模型(LLMs)的推理能力的有效方法。然而,标准代币级剪辑措施仅通过当前代币的条件概率改变政策,而前提条件背景本身是由政策产生的。这种误差在长期响应和政策之外的培训中尤为重要。我们将自动改造作为一个有限的地平线马尔科夫决策过程,并在行为政策采样中获得其绩效差异身份。由此产生的测量变化确定了生成的前提的累积概率,作为每个政策改进术语的准确重量。从信任区域优化的单一改进原理开始,我们在使用准确的基于HTTP调节的基于Chelmatch的政策模型上构建

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate $r-1-\log r$, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Clipping (morphology), Computer science, Autoregressive model, Reinforcement learning, Context (archaeology), Markov decision process

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange