Newest papers

Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

Alyona Nekrasova, Makar Sidorov, Savelii Chezhegov, Alexander Vladimirovich Gasnikov, Margarita M. Preobrazhenskaia, Igor Preobrazhensky, Eduard Gorbunov · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28107 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Gradient fix clipping adalah alat kunci untuk mendapatkan jaminan konvergensi probabilitas tinggi untuk metode stochastic pertama-order di bawah kebisingan ekor berat. Analisis yang ada biasanya mengasumsikan moment sentral terbatas dari urutan $α\in(1,2)$, yang mengecualikan distribusi dengan ekor urutan $u^{-α}$ dan varian dipercepatnya Clip-SSTM pada eksponen kritis. Dalam makalah ini, kita mempertimbangkan optimasi konveks halus di bawah kebisingan ekor berat affine, skala yang dapat tumbuh dengan norma gradient. Kami menghasilkan jaminan konvergensi probabilitas tinggi untuk klipping SGD (Clip-SGD) dan varian Clip-SSTM yang dipercepat di bawah kondisi error $L^$, yang lebih lemah dari batas momen yang sesuai. Untuk $1

Machine translation (Meta M2M100); the English original follows.

Abstract

Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Existing analyses typically assume a bounded central moment of order $α\in(1,2]$, which excludes distributions with tails of order $u^{-α}$ at the critical exponent. In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm. We derive high-probability convergence guarantees for clipped SGD (Clip-SGD) and its accelerated variant Clip-SSTM under a weak-$L^α$ tail condition, which is strictly weaker than the corresponding moment bound. For $1<α<2$, the clipping bounds have the same powers of the clipping level as in the moment case, and both methods attain the additive stochastic rate $K^{-(α-1)/α}$. The gradient-dependent noise contributes a faster-decaying term of order $K^{-2(α-1)/α}$, up to logarithmic factors. At $α=2$, we characterize the additional logarithmic factor on the Lorentz scale and establish its necessity for the additive error at fixed confidence. We also analyze Clip-SGD with prescribed clipping levels and derive finite-horizon guarantees for reaching a neighborhood determined by the clipping bias. For additive noise, matching lower bounds show that this neighborhood is unavoidable, up to constants and the natural error cap. Numerical experiments with Pareto noise illustrate the theoretical results.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Mathematics, Clipping (morphology), Moment (physics), Applied mathematics, Logarithm, Bounded function

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange