Newest papers

Convergence of stochastic gradient descent for learning linear neural networks

Gabin Maxime Nguegnang, Bubacarr Bah, Holger Rauhut, Ulrich Terstiege · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28561 · cited by 0 works (OpenAlex) · open-access PDF

German

Stochastic gradient descent (SGD) ist der Eckpfeiler des modernen tiefen Lernens, aber seine theoretischen Konvergenz-Eigenschaften bleiben in nicht konvexen Einstellungen schlecht verstanden. Um diese Lücke anzugehen, studieren wir SGD-Dynamik in tiefen linearen neuronalen Netzwerken - vereinfachte Modelle, die wichtige nicht-konvexe Eigenschaften behalten, während sie analytisch nachvollziehbar bleiben. Aufbauend auf den jüngsten Ergebnissen für gradient descent, erweitern wir Konvergenz-Garantien auf SGD unter normalisiertem Quadratverlust. Wir etablieren fast sichere Grenzen von SGD-Iteraten. Mit dem Rahmen asymptotischer Pseudotrajectories aus der Stochastic Approximation

Machine translation (Meta M2M100); the English original follows.

Abstract

Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Stochastic gradient descent, Convergence (economics), Lipschitz continuity, Artificial neural network, Mathematics, Mathematical optimization

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange