Orthogonal least squares based fast feature selection for linear classification

• Published in: Pattern recognition Vol. 123; p. 108419
• Main Authors: Zhang, Sikai, Lang, Zi-Qiang
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.03.2022
• Subjects: Canonical correlation analysis; Feature interaction; Feature selection; Linear discriminant analysis; Multi-label; Multivariate time series; Orthogonal least squares; Feature selection; Multivariate time series; Orthogonal least squares; Multi-label; Linear discriminant analysis; Canonical correlation analysis; Feature interaction
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2021.108419

•Based on orthogonal least squares, the novel squared orthogonal correlation coefficients are defined and their relationship with canonical correlation coefficient and linear discriminant analysis is revealed.•An orthogonal least squares based feature selection method is then proposed and it is shown that the method has speed advantages when applied for the greedy search.•A comparison of the proposed method with the mutual information based feature selection methods and the embedded methods shows that the proposed method is always in the top 5 among the 12 candidate methods. An Orthogonal Least Squares (OLS) based feature selection method is proposed for both binomial and multinomial classification. The novel Squared Orthogonal Correlation Coefficient (SOCC) is defined based on Error Reduction Ratio (ERR) in OLS and used as the feature ranking criterion. The equivalence between the canonical correlation coefficient, Fisher’s criterion, and the sum of the SOCCs is revealed, which unveils the statistical implication of ERR in OLS for the first time. It is also shown that the OLS based feature selection method has speed advantages when applied for greedy search. The proposed method is comprehensively compared with the mutual information based feature selection methods and the embedded methods using both synthetic and real world datasets. The results show that the proposed method is always in the top 5 among the 12 candidate methods. Besides, the proposed method can be directly applied to continuous features without discretisation, which is another significant advantage over mutual information based methods.