Nonlinear Regularization Path for Quadratic Loss Support Vector Machines

• Published in: IEEE transactions on neural networks Vol. 22; no. 10; pp. 1613 - 1625
• Main Authors: Karasuyama, M., Takeuchi, I.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.10.2011; Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Applied sciences; Approximation; Approximation algorithms; Approximation methods; Artificial Intelligence; Computer science; control theory; systems; Computers; Data processing. List processing. Character string processing; Equations; Exact sciences and technology; Fasteners; Humans; Inference from stochastic processes; time series analysis; Learning; Machine learning; Mathematical analysis; Mathematical models; Mathematics; Memory organisation. Data processing; Neural Networks (Computer); Nonlinear Dynamics; Nonlinearity; Parametric programming; Piecewise linear approximation; Probability and statistics; rational approximation; Regularization; Sciences and techniques of general use; Software; Software Design; Statistics; Support vector machines; Rational approximation; Breakpoint; Statistical analysis; support vector machines; Linear form; Model selection; Parametric programming; Modeling; Predictor corrector method; Selection problem; Model matching; Quadratic approximation; Vector support machine; Convergence rate; Transmission loss; Rational function; Artificial intelligence; Numerical convergence; Piecewise linearization
• ISSN: 1045-9227; 1941-0093
• DOI: 10.1109/TNN.2011.2164265

Regularization path algorithms have been proposed to deal with model selection problem in several machine learning approaches. These algorithms allow computation of the entire path of solutions for every value of regularization parameter using the fact that their solution paths have piecewise linear form. In this paper, we extend the applicability of regularization path algorithm to a class of learning machines that have quadratic loss and quadratic penalty term. This class contains several important learning machines such as squared hinge loss support vector machine (SVM) and modified Huber loss SVM. We first show that the solution paths of this class of learning machines have piecewise nonlinear form, and piecewise segments between two breakpoints are characterized by a class of rational functions. Then we develop an algorithm that can efficiently follow the piecewise nonlinear path by solving these rational equations. To solve these rational equations, we use rational approximation technique with quadratic convergence rate, and thus, our algorithm can follow the nonlinear path much more precisely than existing approaches such as predictor-corrector type nonlinear-path approximation. We show the algorithm performance on some artificial and real data sets.