Orders of magnitude speed increase in partial least squares feature selection with new simple indexing technique for very tall data sets

• Published in: Journal of chemometrics Vol. 33; no. 11
• Main Authors: Stefansson, Petter, Indahl, Ulf G., Liland, Kristian H., Burud, Ingunn
• Format: Journal Article
• Language: English
• Published: Chichester Wiley Subscription Services, Inc 01.11.2019
• Subjects: Combinatorial analysis; Covariance matrix; feature selection; Indexing; kernel PLS; Least squares; Optimization; PLS; Regression coefficients; subset selection; variable selection
• ISSN: 0886-9383; 1099-128X
• DOI: 10.1002/cem.3141

Feature selection is a challenging combinatorial optimization problem that tends to require a large number of candidate feature subsets to be evaluated before a satisfying solution is obtained. Because of the computational cost associated with estimating the regression coefficients for each subset, feature selection can be an immensely time‐consuming process and is often left inadequately explored. Here, we propose a simple modification to the conventional sequence of calculations involved when fitting a number of feature subsets to the same response data with partial least squares (PLS) model fitting. The modification consists in establishing the covariance matrix for the full set of features by an initial calculation and then deriving the covariance of all subsequent feature subsets solely by indexing into the original covariance matrix. By choosing this approach, which is primarily suitable for tall design matrices with significantly more rows than columns, we avoid redundant (identical) recalculations in the evaluation of different feature subsets. By benchmarking the time required to solve regression problems of various sizes, we demonstrate that the introduced technique outperforms traditional approaches by several orders of magnitude when used in conjunction with PLS modeling. In the supplementary material, we provide code for implementing the concept with kernel PLS regression. We have discovered an indexing technique that can substantially speed up feature selection with partial least squares. The technique consists in establishing the covariance matrix for the full set of features by an initial calculation and then deriving the covariance of all subsequent feature subsets solely by indexing into the original covariance matrix. We prove that this simple concept can speed up feature selection with several orders of magnitude.