Neural collapse with unconstrained features

• Published in: Sampling theory, signal processing, and data analysis Vol. 20; no. 2
• Main Authors: Mixon, Dustin G., Parshall, Hans, Pi, Jianzong
• Format: Journal Article
• Language: English
• Published: Cham Springer International Publishing 01.11.2022
• Subjects: Abstract Harmonic Analysis; Machine Learning; Mathematics; Mathematics and Statistics; Original Article; Signal,Image and Speech Processing; 68T07; Neural collapse; Unconstrained features; Neural networks; 42C15; Equiangular tight frames
• ISSN: 2730-5716; 2730-5724
• DOI: 10.1007/s43670-022-00027-5

Neural collapse is an emergent phenomenon in deep learning that was recently discovered by Papyan, Han and Donoho. We propose a simple unconstrained features model in which neural collapse also emerges empirically. By studying this model, we provide some explanation for the emergence of neural colla...