A CONSTRUCTIVE PREDICTION OF THE GENERALIZATION ERROR ACROSS SCALES

Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit

Research output: Contribution to conferencePaperpeer-review

50 Scopus citations

Abstract

The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

Original languageEnglish
StatePublished - 2020
Event8th International Conference on Learning Representations, ICLR 2020 - Addis Ababa, Ethiopia
Duration: 30 Apr 2020 → …

Conference

Conference8th International Conference on Learning Representations, ICLR 2020
Country/TerritoryEthiopia
CityAddis Ababa
Period30/04/20 → …

Funding

FundersFunder number
National Science FoundationIIS-1447786, CCF-1563880
Air Force Office of Scientific ResearchFA9550-18-1-0054

    Fingerprint

    Dive into the research topics of 'A CONSTRUCTIVE PREDICTION OF THE GENERALIZATION ERROR ACROSS SCALES'. Together they form a unique fingerprint.

    Cite this