Newest papers

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29999 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Reinforcement learning (RL) basado en el corte de estilo PPO se ha convertido en un enfoque eficaz para mejorar las capacidades de razonamiento de los modelos de lenguaje grande (LLM). Sin embargo, las medidas de corte de nivel de token estándar cambian la política sólo a través de la probabilidad condicional del token actual, mientras que el contexto de condicionamiento del prefixo es generado por la política. Este desacuerdo se convierte especialmente en consecuencia en la formación de la respuesta a largo plazo y fuera de la política. Reformulamos la generación autoregresiva como un proceso de decisión de Markov finito-horizonte y derivamos su identidad de diferenciación de rendimiento bajo la muestra de política de comportamiento. El cambio resultante de la medida identifica la relación de probabilidad acumulativa basada en el modelo preexperimental de la

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate $r-1-\log r$, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Clipping (morphology), Computer science, Autoregressive model, Reinforcement learning, Context (archaeology), Markov decision process

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange