Newest papers

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Shiao Xie, Siyu Chen, Jianwei Lv, Xiandong Li, Yujin Wang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26856 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

実証可能な報酬を有する多形修正補正学習(RLVR)は、任務レベルの監督を信頼できるものとしますが、視覚推論における誤りを特定するための限られたガイドラインを提供します。 ポリシー上の自己排除(OPSD)は、この監督を、証拠中心の視覚観点を更新するための特権レベルの教師からの密集したトークンレベルの修正によって補完します。 しかし、結果の修正は、言語的先行者、応答スタイル、または特権レベルの視点自体によって引き起こされる非視覚的な変化と視覚的証拠を一貫して結びつけることができない場合があります。 したがって、直接排除された汚染された修正は、言語的先行者、応

Machine translation (Meta M2M100); the English original follows.

Abstract

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Reinforcement learning, Computer science, Artificial intelligence, Contrast (vision), Human–computer interaction, Visual reasoning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange