Deep Dimension Reduction for Supervised Representation Learning

• Published in: IEEE transactions on information theory Vol. 70; no. 5; pp. 3583 - 3598
• Main Authors: Huang, Jian, Jiao, Yuling, Liao, Xu, Liu, Jin, Yu, Zhou
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.05.2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Cognitive tasks; Conditional independence; Covariance matrices; Data models; Deep learning; Dimensionality reduction; distance covariance; Estimation; f-divergence; Linear programming; Machine learning; neural networks; nonparametric estimation; Reduction; Representation learning; Representations; Vectors
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2023.3340658

The goal of supervised representation learning is to construct effective data representations for prediction. Among all the characteristics of an ideal nonparametric representation of high-dimensional complex data, sufficiency, low dimensionality and disentanglement are some of the most essential ones. We propose a deep dimension reduction approach to learning representations with these characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We formulate the ideal representation learning task as that of finding a nonparametric representation that minimizes an objective function characterizing conditional independence and promoting disentanglement at the population level. We then estimate the target representation at the sample level nonparametrically using deep neural networks. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero. Our extensive numerical experiments using simulated and real benchmark data demonstrate that the proposed methods have better performance than several existing dimension reduction methods and the standard deep learning models in the context of classification and regression.