Why Deep Learning Works: A Manifold Disentanglement Perspective

• Published in: IEEE transaction on neural networks and learning systems Vol. 27; no. 10; pp. 1997 - 2008
• Main Authors: Brahma, Pratik Prabhanjan, Dapeng Wu, Yiyuan She
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Data models; Deep learning; disentanglement; Kernel; Machine learning; manifold learning; Manifolds; Neural networks; Nonhomogeneous media; Principal component analysis; unsupervised feature transformation
• ISSN: 2162-237X; 2162-2388
• DOI: 10.1109/TNNLS.2015.2496947

Deep hierarchical representations of the data have been found out to provide better informative features for several machine learning applications. In addition, multilayer neural networks surprisingly tend to achieve better performance when they are subject to an unsupervised pretraining. The booming of deep learning motivates researchers to identify the factors that contribute to its success. One possible reason identified is the flattening of manifold-shaped data in higher layers of neural networks. However, it is not clear how to measure the flattening of such manifold-shaped data and what amount of flattening a deep neural network can achieve. For the first time, this paper provides quantitative evidence to validate the flattening hypothesis. To achieve this, we propose a few quantities for measuring manifold entanglement under certain assumptions and conduct experiments with both synthetic and real-world data. Our experimental results validate the proposition and lead to new insights on deep learning.