The sample complexity of ERMs in stochastic convex optimization

Daniel Carmon, Roi Livni, Amir Yehudayoff

Research output: Contribution to journalConference articlepeer-review

Abstract

Stochastic convex optimization is one of the most well-studied models for learning in modern machine learning. Nevertheless, a central fundamental question in this setup remained unresolved: how many data points must be observed so that any empirical risk minimizer (ERM) shows good performance on the true population? This question was proposed by Feldman who proved that Ω(dε + ε12 ) data points are necessary (where d is the dimension and " > 0 is the accuracy parameter). Proving an !(dε + ε12 ) lower bound was left as an open problem. In this work we show that in fact Õ(dε + ε12 ) data points are also sufficient. This settles the question and yields a new separation between ERMs and uniform convergence. This sample complexity holds for the classical setup of learning bounded convex Lipschitz functions over the Euclidean unit ball. We further generalize the result and show that a similar upper bound holds for all symmetric convex bodies. The general bound is composed of two terms: (i) a term of the form Õ(dε ) with an inverse-linear dependence on the accuracy parameter, and (ii) a term that depends on the statistical complexity of the class of linear functions (captured by the Rademacher complexity). The proof builds a mechanism for controlling the behavior of stochastic convex optimization problems.

Original languageEnglish
Pages (from-to)3799-3807
Number of pages9
JournalProceedings of Machine Learning Research
Volume238
StatePublished - 2024
Event27th International Conference on Artificial Intelligence and Statistics, AISTATS 2024 - Valencia, Spain
Duration: 2 May 20244 May 2024

Fingerprint

Dive into the research topics of 'The sample complexity of ERMs in stochastic convex optimization'. Together they form a unique fingerprint.

Cite this