Resampling methods for quality assessment of classifier performance and optimal number of features

• Published in: Signal processing Vol. 93; no. 11; pp. 2956 - 2968
• Main Authors: Fandos, Raquel, Debes, Christian, Zoubir, Abdelhak M.
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.11.2013; Elsevier
• Subjects: Algorithms; Applied sciences; Bootstrap; Classification; Classifier design and evaluation; Classifiers; Confidence intervals; Detection, estimation, filtering, equalization, prediction; Exact sciences and technology; Feature evaluation and selection; Information, signal and communications theory; Optimal dimensionality; Optimization; Pattern recognition; Quality assessment; Resampling; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Telecommunications and information theory; Bootstrap; Feature evaluation and selection; Optimal dimensionality; Pattern recognition; Resampling; Classifier design and evaluation; Performance evaluation; Automatic classification; Dimensionality; Probabilistic approach; Probability distribution; Algorithm; Signal classification; Confidence interval; Bootstrapping; Quality control; Testing equipment; Database; Signal processing; Feature extraction; Resampling method; Figure of merit
• DOI: 10.1016/j.sigpro.2013.05.004
• ISSN: 0165-1684

We address two fundamental design issues of a classification system: the choice of the classifier and the dimensionality of the optimal feature subset. Resampling techniques are applied to estimate both the probability distribution of the misclassification rate (or any other figure of merit of a classifier) subject to the size of the feature set, and the probability distribution of the optimal dimensionality given a classification system and a misclassification rate. The latter allows for the estimation of confidence intervals for the optimal feature set size. Based on the former, a quality assessment for the classifier performance is proposed. Traditionally, the comparison of classification systems is accomplished for a fixed feature set. However, a different set may provide different results. The proposed method compares the classifiers independently of any pre-selected feature set. The algorithms are tested on 80 sets of synthetic examples and six standard databases of real data. The simulated data results are verified by an exhaustive search of the optimum and by two feature selection algorithms for the real data sets. •Novel resampling based method.•Choice of the best classifier among set of candidates.•Estimation of the optimal feature set dimensionality.•Algorithm tested on synthetic and real data.