Newest papers

Near Minimax Rate for Quantile Distributional Reinforcement Learning

Zijie Cheng, Zhihua Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29595 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

在这篇论文中,我们研究了在温斯特斯坦距离下对量子分布实例的高效分布政策评估,在该政策下回报分布是由量子贝尔曼的固定风险准确点 $\bmη_m$ 确定的。我们研究了估计 $\bmη_m$ 在温斯特丹距离下对量子分布实例的高效分布的统计准确性。假设访问一个生成模型,我们从一个经验主义的马尔科夫决策过程中构建一个量子固定点风险估计器 $\bmη_m$,并在温斯特丹距离下确立了非常见的本地错误界限。当奖励密度均从零边缘化时,我们获得了对所有分辨率和所有温斯特丹分辨

Machine translation (Meta M2M100); the English original follows.

Abstract

In this paper, we study quantile distributional policy evaluation, where the return distribution under a given policy is approximated by the quantile Bellman fixed point $\bmη_m$. We investigate the statistical accuracy of estimating $\bmη_m$ across quantile resolutions under Wasserstein distances. Assuming access to a generative model, we construct a quantile fixed point estimator $\bmη_m^{(n)}$ from an empirical Markov decision process and establish non-asymptotic error bounds under Wasserstein distances. When reward densities are uniformly bounded away from zero, we obtain a root-$n$ high-probability bound simultaneously over all resolutions $m$ and all Wasserstein orders $1\le p\le\infty$. When reward densities vanish at the support endpoints, we show that continuity and endpoint monotonicity suffice to preserve the same resolution-independent rate for $1\le p\le2$. For $p>2$, we establish a root-$n$ bound with coefficient $m^{1/2-1/p}$, together with a slower, resolution-independent $n^{-1/p}$ bound for every finite $p>2$. We further obtain resolution-independent $W_\infty$ consistency governed by the continuity modulus of reward quantile functions. To investigate the sharpness of these bounds, we characterize the exact local asymptotic minimax risk at each fixed resolution as the expected Wasserstein-coordinate norm of the efficient Gaussian limit and prove that our estimator attains it. We then examine the resolution dependence of this optimal local risk. A regular one-state instance has resolution-independent local risk constants, whereas a single endpoint-vanishing two-state instance exhibits growth of order $m^{1/2-1/p}$, up to logarithmic factors along a resolution subsequence, for every $p>2$.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mathematics, Quantile, Minimax, Estimator, Wasserstein metric, Bounded function

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange