Adaptive Kernel Methods Using the Balancing Principle

• Published in: Foundations of computational mathematics Vol. 10; no. 4; pp. 455 - 479
• Main Authors: De Vito, E., Pereverzyev, S., Rosasco, L.
• Format: Journal Article
• Language: English
• Published: New York Springer-Verlag 01.08.2010; Springer; Springer Nature B.V
• Subjects: Algorithms; Analysis; Applications of Mathematics; Computer Science; Economics; Kernels; Learning; Learning theory; Linear and Multilinear Algebras; Math Applications in Computer Science; Mathematical analysis; Mathematical models; Mathematics; Mathematics and Statistics; Matrix Theory; Methods; Numerical Analysis; Regularization; Strategy; Theory; 68T05; Inverse Problems; 68Q32; Learning Theory; Model Selection; Adaptive Regularization
• ISSN: 1615-3375; 1615-3383
• DOI: 10.1007/s10208-010-9064-2

The regularization parameter choice is a fundamental problem in Learning Theory since the performance of most supervised algorithms crucially depends on the choice of one or more of such parameters. In particular a main theoretical issue regards the amount of prior knowledge needed to choose the regularization parameter in order to obtain good learning rates. In this paper we present a parameter choice strategy, called the balancing principle, to choose the regularization parameter without knowledge of the regularity of the target function. Such a choice adaptively achieves the best error rate. Our main result applies to regularization algorithms in reproducing kernel Hilbert space with the square loss, though we also study how a similar principle can be used in other situations. As a straightforward corollary we can immediately derive adaptive parameter choices for various kernel methods recently studied. Numerical experiments with the proposed parameter choice rules are also presented.