Feature Learning in Deep Classifiers through Intermediate Neural Collapse

Akshay Rangamani*, Marius Lindegaard, Tomer Galanti, Tomaso Poggio

*Corresponding author for this work

Research output: Contribution to journalConference articlepeer-review


In this paper, we conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer's weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories (Willshaw et al., 1969).

Original languageEnglish
Pages (from-to)28729-28745
Number of pages17
JournalProceedings of Machine Learning Research
StatePublished - 2023
Externally publishedYes
Event40th International Conference on Machine Learning, ICML 2023 - Honolulu, United States
Duration: 23 Jul 202329 Jul 2023


Dive into the research topics of 'Feature Learning in Deep Classifiers through Intermediate Neural Collapse'. Together they form a unique fingerprint.

Cite this