Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

Research output: Contribution to conferencePaperpeer-review

Abstract

Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can be used to accelerate model development. In this work, we perform such an empirical investigation across a wide range of language understanding tasks, starting from models with as few as 10K parameters, and evaluate downstream performance across 9 language understanding tasks. We find that scaling laws emerge at finetuning time in some NLP tasks, and that they can also be exploited for debugging convergence when training large models. Moreover, for tasks where scaling laws exist, they can be used to predict the performance of larger models, which enables effective model selection. However, revealing scaling laws requires careful hyperparameter tuning and multiple runs for the purpose of uncertainty estimation, which incurs additional overhead, partially offsetting the computational benefits.

Original languageEnglish
Pages7383-7400
Number of pages18
StatePublished - 2022
Event2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Abu Dhabi, United Arab Emirates
Duration: 7 Dec 202211 Dec 2022

Conference

Conference2022 Findings of the Association for Computational Linguistics: EMNLP 2022
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period7/12/2211/12/22

Funding

FundersFunder number
European Union Horizons 2020 research and innovation programmeDELPHI 802800
Yandex Initiative for Machine Learning
Blavatnik Family Foundation
European Research Council
Israel Science Foundation2486/21

    Fingerprint

    Dive into the research topics of 'Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments'. Together they form a unique fingerprint.

    Cite this