TY - JOUR
T1 - NEAREST CLASS-CENTER SIMPLIFICATION THROUGH INTERMEDIATE LAYERS
AU - Ben-Shaul, Ido
AU - Dekel, Shai
N1 - Publisher Copyright:
© 2022 Proceedings of Machine Learning Research. All rights reserved.
PY - 2022
Y1 - 2022
N2 - Recent advances in neural network theory have introduced geometric properties that occur during training, past the Interpolation Threshold- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the network, and emphasize the innerworkings of Nearest Class-Center Mismatch inside a deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, yielding improvements in both train metrics and generalization.
AB - Recent advances in neural network theory have introduced geometric properties that occur during training, past the Interpolation Threshold- where the training error reaches zero. We inquire into the phenomena coined Neural Collapse in the intermediate layers of the network, and emphasize the innerworkings of Nearest Class-Center Mismatch inside a deepnet. We further show that these processes occur both in vision and language model architectures. Lastly, we propose a Stochastic Variability-Simplification Loss (SVSL) that encourages better geometrical features in intermediate layers, yielding improvements in both train metrics and generalization.
UR - http://www.scopus.com/inward/record.url?scp=85163614007&partnerID=8YFLogxK
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.conferencearticle???
AN - SCOPUS:85163614007
SN - 2640-3498
VL - 196
SP - 37
EP - 47
JO - Proceedings of Machine Learning Research
JF - Proceedings of Machine Learning Research
T2 - ICML Workshop on Topology, Algebra, and Geometry in Machine Learning, TAG:ML 2022
Y2 - 20 July 2022
ER -