Dimensionality reduction by unsupervised regression

• Published in: 2008 IEEE Conference on Computer Vision and Pattern Recognition pp. 1 - 8
• Main Authors: Carreira-Perpinan, M.A., Zhengdong Lu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2008
• Subjects: Backpropagation; Laplace equations; Learning systems; Maximum likelihood estimation; Neural networks; Parameter estimation; Principal component analysis; Prototypes; Spirals
• ISBN: 9781424422425; 1424422426
• ISSN: 1063-6919
• DOI: 10.1109/CVPR.2008.4587666

We consider the problem of dimensionality reduction, where given high-dimensional data we want to estimate two mappings: from high to low dimension (dimensionality reduction) and from low to high dimension (reconstruction). We adopt an unsupervised regression point of view by introducing the unknown low-dimensional coordinates of the data as parameters, and formulate a regularised objective functional of the mappings and low-dimensional coordinates. Alternating minimisation of this functional is straightforward: for fixed low-dimensional coordinates, the mappings have a unique solution; and for fixed mappings, the coordinates can be obtained by finite-dimensional non-linear minimisation. Besides, the coordinates can be initialised to the output of a spectral method such as Laplacian eigenmaps. The model generalises PCA and several recent methods that learn one of the two mappings but not both; and, unlike spectral methods, our model provides out-of-sample mappings by construction. Experiments with toy and real-world problems show that the model is able to learn mappings for convoluted manifolds, avoiding bad local optima that plague other methods.