Batches Stabilize the Minimum Norm Risk in High-Dimensional Overparametrized Linear Regression

Shahar Stein Ioushua, Inbar Hasidim, Ofer Shayevitz, Meir Feder

Research output: Contribution to journalArticlepeer-review

Abstract

Learning algorithms that divide the data into batches are prevalent in many machine-learning applications, typically offering useful trade-offs between computational efficiency and performance. In this paper, we examine the benefits of batch-partitioning through the lens of a minimum-norm overparametrized linear regression model with isotropic Gaussian features. We suggest a natural small-batch version of the minimum-norm estimator and derive bounds on its quadratic risk. We then characterize the optimal batch size and show it is inversely proportional to the noise level, as well as to the overparametrization ratio. In contrast to minimum-norm, our estimator admits a stable risk behavior that is monotonically increasing in the overparametrization ratio, eliminating both the blowup at the interpolation point and the double-descent phenomenon. We further show that shrinking the batch minimum-norm estimator by a factor equal to the Weiner coefficient further stabilizes it and results in lower quadratic risk in all settings. Interestingly, we observe that the implicit regularization offered by the batch partition is partially explained by feature overlap between the batches. Our bound is derived via a novel combination of techniques, in particular normal approximation in the Wasserstein metric of noisy projections over random subspaces.

Original languageEnglish
Pages (from-to)1
Number of pages1
JournalIEEE Transactions on Information Theory
DOIs
StateAccepted/In press - 2024

Funding

FundersFunder number
Israel Science Foundation819/20, 1766/22

    Keywords

    • Linear matrix inequalities
    • Linear regression
    • Partitioning algorithms
    • Servers
    • Signal to noise ratio
    • Task analysis
    • Vectors

    Fingerprint

    Dive into the research topics of 'Batches Stabilize the Minimum Norm Risk in High-Dimensional Overparametrized Linear Regression'. Together they form a unique fingerprint.

    Cite this