Newest papers

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29999 · cited by 0 works (OpenAlex) · open-access PDF

Russian

Reinforcement learning (RL) на основе клиппирования в стиле PPO стал эффективным подходом к улучшению возможностей рассуждений больших языковых моделей (LLM). Однако стандартные меры клиппирования на уровне токенов меняют политику только через условную вероятность текущего токена, в то время как префикс-условный контекст сам по себе генерируется политикой. Это несоответствие становится особенно следственным в долгосрочном и внеполитическом обучении. Мы переформулируем авторегрессивное поколение как конечный горизонтальный процесс решения Маркова и извлекаем его идентичность с разницей в производительности при выборке поведения-политики. Полученное изменение меры идентифицирует совокупное соотношение вероятности генерируемого префикса как точную вес для каждого термина

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate $r-1-\log r$, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Clipping (morphology), Computer science, Autoregressive model, Reinforcement learning, Context (archaeology), Markov decision process

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange