Completely Lazy Learning

• Published in: IEEE transactions on knowledge and data engineering Vol. 22; no. 9; pp. 1274 - 1285
• Main Authors: Garcia, Eric K, Feldman, Sergey, Gupta, Maya R, Srivastava, Santosh
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.09.2010; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Applied sciences; Artificial intelligence; Bayesian estimation; Bayesian methods; Benchmark testing; Classification; Classifiers; Computer science; control theory; systems; Costs; cross validation; Data processing. List processing. Character string processing; Discriminant analysis; Error analysis; Exact sciences and technology; Feedback; Lazy learning; Learning; Learning and adaptive systems; local learning; Memory organisation. Data processing; Nearest neighbor searches; Plant populations; Preprocessing; quadratic discriminant analysis; Software; Studies; Support vector machine classification; Support vector machines; Training; Training data; Trains; Uncertainty; Bayes estimation; Nearest neighbour; Local search; Lazy learning; Discriminant analysis; Bayesian estimation; Statistical analysis; Locality; Cross validation; local learning; Adaptive method; Hyperplane; Vector support machine; Metric; quadratic discriminant analysis
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2009.159

Local classifiers are sometimes called lazy learners because they do not train a classifier until presented with a test sample. However, such methods are generally not completely lazy because the neighborhood size k (or other locality parameter) is usually chosen by cross validation on the training set, which can require significant preprocessing and risks overfitting. We propose a simple alternative to cross validation of the neighborhood size that requires no preprocessing: instead of committing to one neighborhood size, average the discriminants for multiple neighborhoods. We show that this forms an expected estimated posterior that minimizes the expected Bregman loss with respect to the uncertainty about the neighborhood choice. We analyze this approach for six standard and state-of-the-art local classifiers, including discriminative adaptive metric kNN (DANN), a local support vector machine (SVM-KNN), hyperplane distance nearest neighbor (HKNN), and a new local Bayesian quadratic discriminant analysis (local BDA). The empirical effectiveness of this technique versus cross validation is confirmed with experiments on seven benchmark data sets, showing that similar classification performance can be attained without any training.