Tight Risk Bounds for Gradient Descent on Separable Data

Matan Schliserman*, Tomer Koren

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review

1 Scopus citations

Abstract

We study the generalization properties of unregularized gradient methods applied to separable linear classification-a setting that has received considerable attention since the pioneering work of Soudry et al. [14]. We establish tight upper and lower (population) risk bounds for gradient descent in this setting, for any smooth loss function, expressed in terms of its tail decay rate. Our bounds take the form Θ(Equation presented), where (Equation presented) is the number of gradient steps, (Equation presented) is size of the training set, (Equation presented) is the data margin, and (Equation presented) is a complexity term that depends on the tail decay rate of the loss function (and on (Equation presented)). Our upper bound greatly improves the existing risk bounds due to Shamir [13] and Schliserman and Koren [12], that either applied to specific loss functions or imposed extraneous technical assumptions, and applies to virtually any convex and smooth loss function. Our risk lower bound is the first in this context and establish the tightness of our general upper bound for any given tail decay rate and in all parameter regimes. The proof technique used to show these results is also markedly simpler compared to previous work, and is straightforward to extend to other gradient methods; we illustrate this by providing analogous results for Stochastic Gradient Descent.

Original languageEnglish
JournalAdvances in Neural Information Processing Systems
Volume36
StatePublished - 2023
Event37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States
Duration: 10 Dec 202316 Dec 2023

Funding

FundersFunder number
Blavatnik Family Foundation
Aegis Foundation
European Commission
European Research Council101078075
European Research Council
Israel Science Foundation2549/19
Israel Science Foundation

    Fingerprint

    Dive into the research topics of 'Tight Risk Bounds for Gradient Descent on Separable Data'. Together they form a unique fingerprint.

    Cite this