Large-Scale Nyström Kernel Matrix Approximation Using Randomized SVD

• Published in: IEEE transaction on neural networks and learning systems Vol. 26; no. 1; pp. 152 - 164
• Main Authors: Mu Li, Wei Bi, Kwok, James T., Bao-Liang Lu
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithms; Approximation; Approximation algorithms; Approximation methods; Computation; Decomposition; Distributed computing; Eigenvalues; graphics processor; Kernel; Kernels; large-scale learning; low-rank matrix approximation; Mathematical analysis; Matrix decomposition; Neural networks; Nyström method; randomized SVD; Time complexity; large-scale learning; Nyström method; randomized SVD; graphics processor; low-rank matrix approximation; Distributed computing
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2014.2359798

The Nyström method is an efficient technique for the eigenvalue decomposition of large kernel matrices. However, to ensure an accurate approximation, a sufficient number of columns have to be sampled. On very large data sets, the singular value decomposition (SVD) step on the resultant data submatrix can quickly dominate the computations and become prohibitive. In this paper, we propose an accurate and scalable Nyström scheme that first samples a large column subset from the input matrix, but then only performs an approximate SVD on the inner submatrix using the recent randomized low-rank matrix approximation algorithms. Theoretical analysis shows that the proposed algorithm is as accurate as the standard Nyström method that directly performs a large SVD on the inner submatrix. On the other hand, its time complexity is only as low as performing a small SVD. Encouraging results are obtained on a number of large-scale data sets for low-rank approximation. Moreover, as the most computational expensive steps can be easily distributed and there is minimal data transfer among the processors, significant speedup can be further obtained with the use of multiprocessor and multi-GPU systems.