Newest papers

Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Weiqin Chen, Xinlong Du, Gesualdo Scutari, Harsha Honnappa · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28977 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

在情境内增强行为学习(ICRL)利用大型语言模型的令人印象深刻的背景学习能力,以零打一般化到新的,未见的任务,在结论时间没有任何参数更新。在许多方法中,决策预先训练的变换器(DPT)特别吸引其理论基础:在理想化的假设下,如一个足够表达式的模型和无限的预训练数据,DPT可以执行巴耶西后样本。这种特征在实践中是否存活仍然是开放的。因此,我们问: \textit{is ICRL,具体地说是DPT和我们提出的DPT变种,一个巴耶西后样本在实践中?} 接近这个问题,我们建立了两个必要的条件,巴耶西后样本必须满足: \{sufficient line-statistical invariance}

Machine translation (Meta M2M100); the English original follows.

Abstract

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Bayesian probability, Posterior probability, Machine learning, Bayesian inference, Artificial intelligence, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange