## Abstract

We consider the problem of designing uniformly stable first-order optimization algorithms for empirical risk minimization. Uniform stability is often used to obtain generalization error bounds for optimization algorithms, and we are interested in a general approach to achieve it. For Euclidean geometry, we suggest a black-box conversion which given a smooth optimization algorithm, produces a uniformly stable version of the algorithm while maintaining its convergence rate up to logarithmic factors. Using this reduction we obtain a (nearly) optimal algorithm for smooth optimization with convergence rate O^{e}(1/T^{2}) and uniform stability O(T^{2}/n), resolving an open problem of Chen et al. (2018); Attia and Koren (2021). For more general geometries, we develop a variant of Mirror Descent for smooth optimization with convergence rate O^{e}(1/T) and uniform stability O(T/n), leaving open the question of devising a general conversion method as in the Euclidean case.

Original language | English |
---|---|

Pages (from-to) | 3313-3332 |

Number of pages | 20 |

Journal | Proceedings of Machine Learning Research |

Volume | 178 |

State | Published - 2022 |

Event | 35th Conference on Learning Theory, COLT 2022 - London, United Kingdom Duration: 2 Jul 2022 → 5 Jul 2022 |

### Funding

Funders | Funder number |
---|---|

Deutsch Foundation | |

Yandex Initiative in Machine Learning | |

Blavatnik Family Foundation | |

Israel Science Foundation | 2549/19 |