Infinitesimal gradient boosting

• Published in: Stochastic processes and their applications Vol. 170; p. 104310
• Main Authors: Dombry, Clément, Duchamps, Jean-Jil
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.04.2024
• Subjects: Convergence of Markov processes; Gradient boosting; Softmax regression tree; Vanishing-learning-rate asymptotic; 62G05; primary 60F17; Softmax regression tree; Vanishing-learning-rate asymptotic; secondary 60J20; Convergence of Markov processes; Gradient boosting
• ISSN: 0304-4149; 1879-209X
• DOI: 10.1016/j.spa.2024.104310

We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learning rate tends to zero and the number of gradient trees is rescaled accordingly. For this purpose, we introduce a new class of randomized regression trees bridging totally randomized trees and Extra Trees and using a softmax distribution for binary splitting. Our main result is the convergence of the associated stochastic algorithm and the characterization of the limiting procedure as the unique solution of a nonlinear ordinary differential equation in a infinite dimensional function space. Infinitesimal gradient boosting defines a smooth path in the space of continuous functions along which the training error decreases, the residuals remain centered and the total variation is well controlled.