Speeding Up Greedy Forward Selection for Regularized Least-Squares

• Published in: 2010 International Conference on Machine Learning and Applications pp. 325 - 330
• Main Authors: Pahikkala, Tapio, Airola, Antti, Salakoski, Tapio
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2010
• Subjects: Computational complexity; feature selection; greedy algorithm; least-squares support vector machine; Machine learning; Machine learning algorithms; Prediction algorithms; regularized least-squares; ridge regression; Training; Vectors
• ISBN: 1424492114; 9781424492114
• DOI: 10.1109/ICMLA.2010.55

We propose a novel algorithm for greedy forward feature selection for regularized least-squares (RLS) regression and classification, also known as the least-squares support vector machine or ridge regression. The algorithm, which we call greedy RLS, starts from the empty feature set, and on each iteration adds the feature whose addition provides the best leave-one-out cross-validation performance. Our method is considerably faster than the previously proposed ones, since its time complexity is linear in the number of training examples, the number of features in the original data set, and the desired size of the set of selected features. Therefore, as a side effect we obtain a new training algorithm for learning sparse linear RLS predictors which can be used for large scale learning. This speed is possible due to matrix calculus based short-cuts for leave-one-out and feature addition. We experimentally demonstrate the scalability of our algorithm compared to previously proposed implementations.