The Early Restart Algorithm

• Published in: Neural computation Vol. 12; no. 6; pp. 1303 - 1312
• Main Authors: Magdon-Ismail, Malik, Atiya, Amir F.
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.06.2000
• Subjects: Algorithmics. Computability. Computer arithmetics; Algorithms; Applied sciences; Artificial intelligence; Computer science; control theory; systems; Connectionism. Neural networks; Exact sciences and technology; Linear Models; Neural Networks (Computer); Theoretical computing; Probability density; Backpropagation algorithm; Steepest descent method; Neural network; Learning algorithm; Algorithm; Convergence
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/089976600300015376

Consider an algorithm whose time to convergence is unknown (because of some random element in the algorithm, such as a random initial weight choice for neural network training). Consider the following strategy. Run the algorithm for a specific time . If it has not converged by time , cut the run short and rerun it from the start (repeat the same strategy for every run). This so-called restart mechanism has been proposed by Fahlman (1988) in the context of backpropagation training. It is advantageous in problems that are prone to local minima or when there is a large variability in convergence time from run to run, and may lead to a speed-up in such cases. In this article, we analyze theoretically the restart mechanism, and obtain conditions on the probability density of the convergence time for which restart will improve the expected convergence time. We also derive the optimal restart time. We apply the derived formulas to several cases, including steepest-descent algorithms.