Newest papers

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26856 · cited by 0 works (OpenAlex) · open-access PDF

Korean

다중 형태의 시각적 증거에서 오류를 현지화하는 데 유한한 지침을 제공합니다. 정책 자체 증거 (OPSD)는 증거 중심의 시각적 견해를 업데이트 할 때 특권 수준의 교사로부터 밀접한 토큰 수준의 수정으로이 감독을 보완합니다. 그러나 결과적인 수정은 언어적 프리어, 응답 스타일 또는 특권적 견해 자체에 의해 유도되는 비시각적 변동과 시각적 증거를 혼동시킬 수 있습니다. 우리는이 현상을 특권적 인 교정 오염으로 공식화합니다 : 토큰 수준의 수정은 증거 방향을 업데이트 할 때 오염됩니다. 결과적인 수정은 특권적 인 시각적 증거에 일관되게 기인 할 수 없습니다. 직접 증거 된 오염 수정은 따라서 언어 수준의 오류

Machine translation (Meta M2M100); the English original follows.

Abstract

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Reinforcement learning, Computer science, Artificial intelligence, Contrast (vision), Human–computer interaction, Visual reasoning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange