SGD Generalizes Better Than GD (And Regularization Doesn't Help)

Idan Amir; Tomer Koren; Roi Livni

SGD Generalizes Better Than GD (And Regularization Doesn't Help)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We give a new separation result between the generalization performance of stochastic gradient descent (SGD) and of full-batch gradient descent (GD) in the fundamental stochastic convex optimization model. While for SGD it is well-known that O(1/ϵ²) iterations suffice for obtaining a solution with ϵ excess expected risk, we show that with the same number of steps GD may overfit and emit a solution with Ω(1) generalization error. Moreover, we show that in fact Ω(1/ϵ⁴) iterations are necessary for GD to match the generalization performance of SGD, which is also tight due to recent work by Bassily et al. (2020). We further discuss how regularizing the empirical risk minimized by GD essentially does not change the above result, and revisit the concepts of stability, implicit bias and the role of the learning algorithm in generalization.

Original language	English
Title of host publication	Proceedings of Thirty Fourth Conference on Learning Theory
Editors	Mikhail Belkin, Samory Kpotufe
Publisher	PMLR
Pages	63-92
Number of pages	30
State	Published - 2021
Event	34th Annual Conference on Learning Theory, COLT 2021 - Boulder, United States Duration: 15 Aug 2021 → 19 Aug 2021 Conference number: 34

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
Volume	134
ISSN (Electronic)	2640-3498

Conference

Conference	34th Annual Conference on Learning Theory, COLT 2021
Abbreviated title	COLT 2021
Country/Territory	United States
City	Boulder
Period	15/08/21 → 19/08/21

Access to Document

Cite this

@inproceedings{69faf0fce18c42d3a0ca9ad38445dc71,

title = "SGD Generalizes Better Than GD (And Regularization Doesn't Help)",

abstract = "We give a new separation result between the generalization performance of stochastic gradient descent (SGD) and of full-batch gradient descent (GD) in the fundamental stochastic convex optimization model. While for SGD it is well-known that O(1/ϵ2) iterations suffice for obtaining a solution with ϵ excess expected risk, we show that with the same number of steps GD may overfit and emit a solution with Ω(1) generalization error. Moreover, we show that in fact Ω(1/ϵ4) iterations are necessary for GD to match the generalization performance of SGD, which is also tight due to recent work by Bassily et al. (2020). We further discuss how regularizing the empirical risk minimized by GD essentially does not change the above result, and revisit the concepts of stability, implicit bias and the role of the learning algorithm in generalization.",

author = "Idan Amir and Tomer Koren and Roi Livni",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.; 34th Annual Conference on Learning Theory, COLT 2021 ; Conference date: 15-08-2021 Through 19-08-2021",

year = "2021",

language = "אנגלית",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "63--92",

editor = "Mikhail Belkin and Samory Kpotufe",

booktitle = "Proceedings of Thirty Fourth Conference on Learning Theory",

}

Amir, I, Koren, T & Livni, R 2021, SGD Generalizes Better Than GD (And Regularization Doesn't Help). in M Belkin & S Kpotufe (eds), Proceedings of Thirty Fourth Conference on Learning Theory. Proceedings of Machine Learning Research, vol. 134, PMLR, pp. 63-92, 34th Annual Conference on Learning Theory, COLT 2021, Boulder, Colorado, United States, 15/08/21. <https://dblp.org/db/conf/colt/colt2021.html#AmirKL21>

SGD Generalizes Better Than GD (And Regularization Doesn't Help). / Amir, Idan; Koren, Tomer ; Livni, Roi.
Proceedings of Thirty Fourth Conference on Learning Theory. ed. / Mikhail Belkin; Samory Kpotufe. PMLR, 2021. p. 63-92 (Proceedings of Machine Learning Research; Vol. 134).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - SGD Generalizes Better Than GD (And Regularization Doesn't Help)

AU - Amir, Idan

AU - Koren, Tomer

AU - Livni, Roi

N1 - Conference code: 34

PY - 2021

Y1 - 2021

N2 - We give a new separation result between the generalization performance of stochastic gradient descent (SGD) and of full-batch gradient descent (GD) in the fundamental stochastic convex optimization model. While for SGD it is well-known that O(1/ϵ2) iterations suffice for obtaining a solution with ϵ excess expected risk, we show that with the same number of steps GD may overfit and emit a solution with Ω(1) generalization error. Moreover, we show that in fact Ω(1/ϵ4) iterations are necessary for GD to match the generalization performance of SGD, which is also tight due to recent work by Bassily et al. (2020). We further discuss how regularizing the empirical risk minimized by GD essentially does not change the above result, and revisit the concepts of stability, implicit bias and the role of the learning algorithm in generalization.

AB - We give a new separation result between the generalization performance of stochastic gradient descent (SGD) and of full-batch gradient descent (GD) in the fundamental stochastic convex optimization model. While for SGD it is well-known that O(1/ϵ2) iterations suffice for obtaining a solution with ϵ excess expected risk, we show that with the same number of steps GD may overfit and emit a solution with Ω(1) generalization error. Moreover, we show that in fact Ω(1/ϵ4) iterations are necessary for GD to match the generalization performance of SGD, which is also tight due to recent work by Bassily et al. (2020). We further discuss how regularizing the empirical risk minimized by GD essentially does not change the above result, and revisit the concepts of stability, implicit bias and the role of the learning algorithm in generalization.

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

T3 - Proceedings of Machine Learning Research

SP - 63

EP - 92

BT - Proceedings of Thirty Fourth Conference on Learning Theory

A2 - Belkin, Mikhail

A2 - Kpotufe, Samory

PB - PMLR

T2 - 34th Annual Conference on Learning Theory, COLT 2021

Y2 - 15 August 2021 through 19 August 2021

ER -

SGD Generalizes Better Than GD (And Regularization Doesn't Help)

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this