Multi-resolution boosting for classification and regression problems

• Published in: Knowledge and information systems Vol. 29; no. 2; pp. 435 - 456
• Main Authors: Reddy, Chandan K., Park, Jin-Hyeong
• Format: Journal Article
• Language: English
• Published: London Springer-Verlag 01.11.2011; Springer; Springer Nature B.V
• Subjects: Additives; Algorithms; Analogies; Applied sciences; Artificial intelligence; Classification; Computer Science; Computer science; control theory; systems; Data mining; Data Mining and Knowledge Discovery; Data processing. List processing. Character string processing; Database Management; Exact sciences and technology; Information Storage and Retrieval; Information systems; Information Systems and Communication Service; Information Systems Applications (incl.Internet); Investigations; IT in Business; Machine learning; Mathematical models; Memory organisation. Data processing; Neural networks; Optimization; Regression; Regular Paper; Software; Studies; Success; Gaussian kernel; Classification; Additive models; Regression; Boosting; Multi-resolution; Weak learner; Data analysis; Automatic classification; Regression analysis; Data mining; Modeling; Multiresolution analysis; Aggregate model; Radial basis function; Loss function; Gaussian process; Weak solution; Supervised learning; Weak leaner; Artificial intelligence
• ISSN: 0219-1377; 0219-3116
• DOI: 10.1007/s10115-010-0358-0

Various forms of additive modeling techniques have been successfully used in many data mining and machine learning–related applications. In spite of their great success, boosting algorithms still suffer from a few open-ended problems that require closer investigation. The efficiency of any additive modeling technique relies significantly on the choice of the weak learners and the form of the loss function. In this paper, we propose a novel multi-resolution approach for choosing the weak learners during additive modeling. Our method applies insights from multi-resolution analysis and chooses the optimal learners at multiple resolutions during different iterations of the boosting algorithms, which are simple yet powerful additive modeling methods. We demonstrate the advantages of this novel framework in both classification and regression problems and show results on both synthetic and real-world datasets taken from the UCI machine learning repository. Though demonstrated specifically in the context of boosting algorithms, our framework can be easily accommodated in general additive modeling techniques. Similarities and distinctions of the proposed algorithm with the popularly used methods like radial basis function networks are also discussed.