Newest papers

Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Weiqin Chen, Xinlong Du, Gesualdo Scutari, Harsha Honnappa · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28977 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Dalam konteks pembelajaran perbaikan perilaku (ICRL) mengeksploitasi kemampuan belajar dalam konteks yang mengesankan dari model bahasa besar untuk generalisasi nol-shot ke tugas-tugas baru, tak terlihat pada waktu inferensi tanpa pembaruan parameter apa pun. Di antara banyak pendekatan, Decision-Pretrained Transformer (DPT) sangat menarik untuk landasan teoritisnya: di bawah asumsi-asumsi yang diidealisasi seperti model ekspresif yang cukup dan data pra-latihan yang tak terbatas, DPT dapat melakukan sampling belakang Bayesian. Apakah karakterisasi ini bertahan dalam praktek tetap terbuka. Oleh karena itu, kami bertanya: \textit{is ICRL, khususnya DPT dan varian DPT kami yang diusulkan, seorang sampler belakang Bayesian dalam praktek?} Untuk mendekati pertanyaan ini, kami menetapkan dua kondisi yang diperlukan

Machine translation (Meta M2M100); the English original follows.

Abstract

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: \textit{is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice?} To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: \textit{sufficient-statistic invariance} and the \textit{martingale property}. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Bayesian probability, Posterior probability, Machine learning, Bayesian inference, Artificial intelligence, Computer science

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange