Newest papers

Convergence of stochastic gradient descent for learning linear neural networks

Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut, Ulrich Terstiege · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28561 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

SGD 是现代深度学习的基石,但其理论融合特性在非凝固式设置中仍然不太了解。为了解决这种差距,我们研究 SGD 动态在深线性神经网络中 - 简化模型,这些模型保持了关键的非凝固性特性,同时仍然可以分析。 基于近期结果的凝固性下降,我们在规律化的平方损失下向 SGD 扩展了凝固性保证。 我们建立了 SGD 迭代的几乎确定的边界性。 使用来自 Stochastic Approximation 理论的无对称性假轨迹的框架,我们的分析建立了 SGD 到温和条件下的关键点的集合(而无需使用 Lipschitz 或我们自己的数据)。

Machine translation (Meta M2M100); the English original follows.

Abstract

Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Stochastic gradient descent, Convergence (economics), Lipschitz continuity, Artificial neural network, Mathematics, Mathematical optimization

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange