Newest papers

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29999 · cited by 0 works (OpenAlex) · open-access PDF

Korean

PPO 스타일의 클리핑을 바탕으로 하는 코드 클리핑은 대형 언어 모델 (LLM)의 이론적 능력을 향상시키는 효과적인 접근법이되었습니다. 그러나 표준 토큰 수준의 클리핑 측정 정책은 현재 토큰의 조건적 확률을 통해서만 변경되며, 프리피스 조건화 컨텍스트 자체는 정책에 의해 생성됩니다. 이 불일치는 특히 긴 응답 및 정책 외부 교육에서 결과적입니다. 우리는 단계적으로 정책 개선 용어를 단계적으로 생성하고 행동 정책 샘플링에서 성능 차이 정체성을 추출합니다. 결과적인 측정의 변화는 생성된 프리피스의 축적 확률 비율을 각 정책 개선 용어의 정확한 중요성으로 식별합니다. 신뢰-지역 지역 최적화의 단계적

Machine translation (Meta M2M100); the English original follows.

Abstract

Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate $r-1-\log r$, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at https://github.com/LiChengxi666/EPIC.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Clipping (morphology), Computer science, Autoregressive model, Reinforcement learning, Context (archaeology), Markov decision process

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange