Abstract
We propose a method that achieves near-optimal rates for smooth stochastic convex optimization and requires essentially no prior knowledge of problem parameters. This improves on prior work which requires knowing at least the initial distance to optimality d0. Our method, U-DOG, combines UniXGrad (Kavis et al. [30]) and DoG (Ivgi et al. [27]) with novel iterate stabilization techniques. It requires only loose bounds on d0 and the noise magnitude, provides high probability guarantees under sub-Gaussian noise, and is also near-optimal in the non-smooth case. Our experiments show consistent, strong performance on convex problems and mixed results on neural network training.
Original language | English |
---|---|
Pages (from-to) | 3257-3324 |
Number of pages | 68 |
Journal | Proceedings of Machine Learning Research |
Volume | 247 |
State | Published - 2024 |
Event | 37th Annual Conference on Learning Theory, COLT 2024 - Edmonton, Canada Duration: 30 Jun 2024 → 3 Jul 2024 |
Funding
Funders | Funder number |
---|---|
Council for Higher Education | |
NSF-BSF | |
Pitt Momentum Funds | |
National Science Foundation | 2239527 |
United States-Israel Binational Science Foundation | 2022663 |
Air Force Office of Scientific Research | #FA955023-1-0242 |
Israel Science Foundation | 2486/21 |
Keywords
- Adaptive
- Parameter-free
- Smooth optimization
- Stochastic convex optimization