Experimental Design for Overparameterized Learning With Application to Single Shot Deep Active Learning

Neta Shoham*, Haim Avron

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

The impressive performance exhibited by modern machine learning models hinges on the ability to train such models on a very large amounts of labeled data. However, since access to large volumes of labeled data is often limited or expensive, it is desirable to alleviate this bottleneck by carefully curating the training set. Optimal experimental design is a well-established paradigm for selecting data point to be labeled so to maximally inform the learning process. Unfortunately, classical theory on optimal experimental design focuses on selecting examples in order to learn underparameterized (and thus, non-interpolative) models, while modern machine learning models such as deep neural networks are overparameterized, and oftentimes are trained to be interpolative. As such, classical experimental design methods are not applicable in many modern learning setups. Indeed, the predictive performance of underparameterized models tends to be variance dominated, so classical experimental design focuses on variance reduction, while the predictive performance of overparameterized models can also be, as is shown in this paper, bias dominated or of mixed nature. In this paper we propose a design strategy that is well suited for overparameterized regression and interpolation, and we demonstrate the applicability of our method in the context of deep learning by proposing a new algorithm for single shot deep active learning.

Original languageEnglish
Pages (from-to)11766-11777
Number of pages12
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume45
Issue number10
DOIs
StatePublished - 1 Oct 2023

Keywords

  • Statistical learning

Fingerprint

Dive into the research topics of 'Experimental Design for Overparameterized Learning With Application to Single Shot Deep Active Learning'. Together they form a unique fingerprint.

Cite this