TY - CONF

T1 - SGD learns over-parameterized networks that provably generalize on linearly separable data

AU - Brutzkus, Alon

AU - Globerson, Amir

AU - Malach, Eran

AU - Shalev-Shwartz, Shai

N1 - Publisher Copyright:
© Learning Representations, ICLR 2018 - Conference Track Proceedings.All right reserved.

PY - 2018

Y1 - 2018

N2 - Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations. Nonetheless, current generalization bounds for neural networks fail to explain this phenomenon. In an attempt to bridge this gap, we study the problem of learning a two-layer over-parameterized neural network, when the data is generated by a linearly separable function. In the case where the network has Leaky ReLU activations and only the first layer is trained, we provide both optimization and generalization guarantees for over-parameterized networks. Specifically, we prove convergence rates of SGD to a global minimum, and provide generalization guarantees for this global minimum that are independent of the network size. Therefore, our result clearly shows that the use of SGD for optimization both finds a global minimum, and avoids overfitting despite the high capacity of the model. This is the first theoretical demonstration that SGD can avoid overfitting, when learning over-specified neural network classifiers.

AB - Neural networks exhibit good generalization behavior in the over-parameterized regime, where the number of network parameters exceeds the number of observations. Nonetheless, current generalization bounds for neural networks fail to explain this phenomenon. In an attempt to bridge this gap, we study the problem of learning a two-layer over-parameterized neural network, when the data is generated by a linearly separable function. In the case where the network has Leaky ReLU activations and only the first layer is trained, we provide both optimization and generalization guarantees for over-parameterized networks. Specifically, we prove convergence rates of SGD to a global minimum, and provide generalization guarantees for this global minimum that are independent of the network size. Therefore, our result clearly shows that the use of SGD for optimization both finds a global minimum, and avoids overfitting despite the high capacity of the model. This is the first theoretical demonstration that SGD can avoid overfitting, when learning over-specified neural network classifiers.

UR - http://www.scopus.com/inward/record.url?scp=85083950113&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontoconference.paper???

AN - SCOPUS:85083950113

T2 - 6th International Conference on Learning Representations, ICLR 2018

Y2 - 30 April 2018 through 3 May 2018

ER -