Uniform Stability for First-Order Empirical Risk Minimization

Amit Attia, Tomer Koren

Research output: Contribution to journalConference articlepeer-review

Abstract

We consider the problem of designing uniformly stable first-order optimization algorithms for empirical risk minimization. Uniform stability is often used to obtain generalization error bounds for optimization algorithms, and we are interested in a general approach to achieve it. For Euclidean geometry, we suggest a black-box conversion which given a smooth optimization algorithm, produces a uniformly stable version of the algorithm while maintaining its convergence rate up to logarithmic factors. Using this reduction we obtain a (nearly) optimal algorithm for smooth optimization with convergence rate Oe(1/T2) and uniform stability O(T2/n), resolving an open problem of Chen et al. (2018); Attia and Koren (2021). For more general geometries, we develop a variant of Mirror Descent for smooth optimization with convergence rate Oe(1/T) and uniform stability O(T/n), leaving open the question of devising a general conversion method as in the Euclidean case.

Original languageEnglish
Pages (from-to)3313-3332
Number of pages20
JournalProceedings of Machine Learning Research
Volume178
StatePublished - 2022
Event35th Conference on Learning Theory, COLT 2022 - London, United Kingdom
Duration: 2 Jul 20225 Jul 2022

Funding

FundersFunder number
Deutsch Foundation
Yandex Initiative in Machine Learning
Blavatnik Family Foundation
Israel Science Foundation2549/19

    Fingerprint

    Dive into the research topics of 'Uniform Stability for First-Order Empirical Risk Minimization'. Together they form a unique fingerprint.

    Cite this