Deterministic Gradient-Descent Learning of Linear Regressions: Adaptive Algorithms, Convergence Analysis and Noise Compensation

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 46; no. 12; pp. 7867 - 7877
• Main Authors: Liu, Kang-Zhi, Gan, Chao
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.12.2024
• Subjects: adaptive learning rate; Convergence; Estimation error; gradient descent; Linear regression; Noise; noise compensation; Noise measurement; Stochastic processes; Vectors; weight convergence
• ISSN: 0162-8828; 1939-3539; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2024.3399312

Weight learning forms a basis for the machine learning and numerous algorithms have been adopted up to date. Most of the algorithms were either developed in the stochastic framework or aimed at minimization of loss or regret functions. Asymptotic convergence of weight learning, vital for good output prediction, was seldom guaranteed for online applications. Since linear regression is the most fundamental component in machine learning, we focus on this model in this paper. Aiming at online applications, a deterministic analysis method is developed based on LaSalle's invariance principle. Convergence conditions are derived for both the first-order and the second-order learning algorithms, without resorting to any stochastic argument. Moreover, the deterministic approach makes it easy to analyze the noise influence. Specifically, adaptive hyperparameters are derived in this framework and their tuning rules disclosed for the compensation of measurement noise. Comparison with four most popular algorithms validates that this approach has a higher learning capability and is quite promising in enhancing the weight learning performance.