Penalized Preimage Learning in Kernel Principal Component Analysis

• Published in: IEEE transactions on neural networks Vol. 21; no. 4; pp. 551 - 570
• Main Authors: Wei-Shi Zheng, JianHuang Lai, Yuen, P.C.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.04.2010; Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Applied sciences; Artificial Intelligence; Computer science; control theory; systems; Computer Simulation; Connectionism. Neural networks; Constraint optimization; Exact sciences and technology; Humans; Image denoising; Image Interpretation, Computer-Assisted - methods; Image reconstruction; Kernel; kernel principal component analysis (KPCA); Kernels; Laplace equations; Learning; Learning and adaptive systems; Lighting; locality preservation; Mathematical analysis; Mathematical models; Mean square error methods; Mutual information; Optimization; Pattern Recognition, Automated; Pattern recognition. Digital image processing. Computational geometry; penalty function; preimage problem; Principal Component Analysis; Turning; Vectors (mathematics); Occlusion; Image processing; Data mining; Modeling; Optimization; Image reconstruction; Mean square error; Penalty function; Facies; Illumination; preimage problem; Ill posed problem; kernel principal component analysis (KPCA); Convexity; Learning algorithm; Kernel; Locality; Image databank; Image restoration; Neural network; Laplacian; Kernel method; Image quality; Luminance; locality preservation; Artificial intelligence; Occultation; Mutual information; Facial expression; Principal component analysis
• DOI: 10.1109/TNN.2009.2039647
• ISSN: 1045-9227; 1941-0093

Finding the preimage of a feature vector in kernel principal component analysis (KPCA) is of crucial importance when KPCA is applied in some applications such as image preprocessing. Since the exact preimage of a feature vector in the kernel feature space, normally, does not exist in the input data space, an approximate preimage is learned and encouraging results have been reported in the last few years. However, it is still difficult to find a "good" estimation of preimage. As estimation of preimage in kernel methods is ill-posed, how to guide the preimage learning for a better estimation is important and still an open problem. To address this problem, a penalized strategy is developed in this paper, where some penalization terms are used to guide the preimage learning process. To develop an efficient penalized technique, we first propose a two-step general framework, in which a preimage is directly modeled by weighted combination of the observed samples and the weights are learned by some optimization function subject to certain constraints. Compared to existing techniques, this would also give advantages in directly turning preimage learning into the optimization of the combination weights. Under this framework, a penalized methodology is developed by integrating two types of penalizations. First, to ensure learning a well-defined preimage, of which each entry is not out of data range, convexity constraint is imposed for learning the combination weights. More insight effects of the convexity constraint are also explored. Second, a penalized function is integrated as part of the optimization function to guide the preimage learning process. Particularly, the weakly supervised penalty is proposed, discussed, and extensively evaluated along with Laplacian penalty and ridge penalty. It could be further interpreted that the learned preimage can preserve some kind of pointwise conditional mutual information. Finally, KPCA with preimage learning is applied on face image data sets in the aspects of facial expression normalization, face image denoising, recovery of missing parts from occlusion, and illumination normalization. Experimental results show that the proposed preimage learning algorithm obtains lower mean square error (MSE) and better visual quality of reconstructed images.