Revisiting the Noise Model of Stochastic Gradient Descent

Barak Battash, Lior Wolf, Ofir Lindenbaum

Research output: Contribution to journalConference articlepeer-review

Abstract

The effectiveness of stochastic gradient descent (SGD) in neural network optimization is significantly influenced by stochastic gradient noise (SGN). Following the central limit theorem, SGN was initially described as Gaussian, but recently Simsekli et al. (2019) demonstrated that the SαS Lévy distribution provides a better fit for the SGN. This assertion was purportedly debunked and rebounded to the Gaussian noise model that had been previously proposed. This study provides robust, comprehensive empirical evidence that SGN is heavy-tailed and is better represented by the SαS distribution. Our experiments include several datasets and multiple models, both discriminative and generative. Furthermore, we argue that different network parameters preserve distinct SGN properties. We develop a novel framework based on a Lévy-driven stochastic differential equation (SDE), where one-dimensional Lévy processes describe each parameter. This leads to a more accurate characterization of the dynamics of SGD around local minima. We use our framework to study SGD properties near local minima; these include the mean escape time and preferable exit directions.

Original languageEnglish
Pages (from-to)4780-4788
Number of pages9
JournalProceedings of Machine Learning Research
Volume238
StatePublished - 2024
Event27th International Conference on Artificial Intelligence and Statistics, AISTATS 2024 - Valencia, Spain
Duration: 2 May 20244 May 2024

Funding

FundersFunder number
Tel Aviv University

    Fingerprint

    Dive into the research topics of 'Revisiting the Noise Model of Stochastic Gradient Descent'. Together they form a unique fingerprint.

    Cite this