Abstract
We study Stochastic Gradient Descent with AdaGrad stepsizes: a popular adaptive (self-tuning) method for first-order stochastic optimization. Despite being well studied, existing analyses of this method suffer from various shortcomings: they either assume some knowledge of the problem parameters, impose strong global Lipschitz conditions, or fail to give bounds that hold with high probability. We provide a comprehensive analysis of this basic method without any of these limitations, in both the convex and non-convex (smooth) cases, that additionally supports a general “affine variance” noise model and provides sharp rates of convergence in both the low-noise and high-noise regimes.
| Original language | English |
|---|---|
| Pages (from-to) | 1147-1171 |
| Number of pages | 25 |
| Journal | Proceedings of Machine Learning Research |
| Volume | 202 |
| State | Published - 2023 |
| Event | 40th International Conference on Machine Learning, ICML 2023 - Honolulu, United States Duration: 23 Jul 2023 → 29 Jul 2023 |
Funding
| Funders | Funder number |
|---|---|
| Adelis Research Fund for Artificial Intelligence | |
| Yandex Initiative for Machine Learning | |
| Blavatnik Family Foundation | |
| Israel Science Foundation | 993/17 |
| Tel Aviv University |