Deep Linear Networks for Matrix Completion-an Infinite Depth Limit

Nadav Cohen, Govind Menon, Zsolt Veraszto

Research output: Contribution to journalArticlepeer-review

Abstract

The deep linear network (DLN) is a model for implicit regularization in gradient based optimization of overparametrized learning architectures. Training the DLN corresponds to a Riemannian gradient flow, where the Riemannian metric is defined by the architecture of the network and the loss function is defined by the learning task. We extend this geometric framework, obtaining explicit expressions for the volume form, including the case when the network has infinite depth. We investigate the link between the Riemannian geometry and the training asymptotics for matrix completion with rigorous analysis and numerics. We propose that under small initialization, implicit regularization is a result of bias towards high state space volume.

Original languageEnglish
Pages (from-to)3208-3232
Number of pages25
JournalSIAM Journal on Applied Dynamical Systems
Volume22
Issue number4
DOIs
StatePublished - 2023

Funding

FundersFunder number
National Science FoundationDMS-2107205
Israel Science Foundation1780/21
Tel Aviv University

    Keywords

    • Riemannian gradient flow
    • deep linear network
    • generalizability
    • implicit regularization
    • matrix completion

    Fingerprint

    Dive into the research topics of 'Deep Linear Networks for Matrix Completion-an Infinite Depth Limit'. Together they form a unique fingerprint.

    Cite this