Newest papers

Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms

Martin Hernandez, Xinyan Qing, Mo Zhou, Yuhua Zhu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28580 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

持续时间增强数据学习寻求在未知动态下以斯托加斯特差异方程驱动的最佳政策,使用在分散时间收集的观察。我们开发了一套政策基层框架,该框架将严格的汇合分析与从分散数据学习相结合。我们首先建立了性能差异LEMMA,并衍生了随机和参数化政策的政策基层方程式。然后我们证明了在未知动态下学习的指数式汇合,该框架与PhiBE结合在一起,该框架构建了从分散的数据转换而来的连续时间替代发电机。 由此产生的政策和非政策模型对演员的批判性错误、批判性错误和学习步骤大小的贡献分离了汇总线。

Machine translation (Meta M2M100); the English original follows.

Abstract

Continuous-time reinforcement learning seeks an optimal policy under unknown dynamics driven by stochastic differential equations using observations collected at discrete times. We develop a policy gradient framework that combines rigorous convergence analysis with learning from discrete data. We first establish a performance difference lemma and derive policy gradient formulas for randomized and parametrized policies. We then prove exponential convergence of an entropy-regularized policy flow to the regularized optimum. For discrete parametrized updates, we derive a convergence bound that explicitly separates the contributions of critic error, policy-regression error, and the learning step size. To enable learning under unknown dynamics, we integrate this framework with PhiBE, which constructs a continuous-time surrogate generator from discrete transition data. The resulting on-policy and off-policy model-free actor-critic algorithms preserve the continuous-time structure of the control problem. Numerical experiments on an entropy-regularized linear-quadratic regulator and a nonlinear wheeled robot demonstrate the performance and stability of the proposed methods with both known and data-driven dynamics.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Convergence (economics), Reinforcement learning, Mathematics, Lemma (botany), Stability (learning theory), Discrete time and continuous time

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange