Convergence and Objective Functions of Some Fault/Noise-Injection-Based Online Learning Algorithms for RBF Networks

• Published in: IEEE transactions on neural networks Vol. 21; no. 6; pp. 938 - 947
• Main Authors: Ho, Kevin I.-J, Chi-Sing Leung, Sum, John
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.06.2010; Institute of Electrical and Electronics Engineers
• Subjects: Additive noise; Additives; Algorithmics. Computability. Computer arithmetics; Algorithms; Applied sciences; Artificial intelligence; Circuit faults; Computer science; control theory; systems; Computer systems performance. Reliability; Connectionism. Neural networks; Convergence; Exact sciences and technology; Fault tolerance; Faults; Field programmable gate arrays; gladyshev theorem; Humans; Learning; Local government; Mean square error methods; Neural networks; Neural Networks (Computer); Noise; objective functions; On-line systems; Online; Online Systems; Radial basis function networks; RBF Networks; Recurrent neural networks; Signal Processing, Computer-Assisted; Software; Theoretical computing; Training; Additive noise; On line; Probabilistic approach; objective functions; Neural network; RBF Networks; Approximation algorithm; Fault injection; Combinatorial optimization; Program verification; Mean square error; Convergence; Radial basis function; Fault tolerance; Multiplicative noise; gladyshev theorem; Objective function; Learning algorithm; Artificial intelligence; Fault tolerant system
• ISSN: 1045-9227; 1941-0093; 1941-0093
• DOI: 10.1109/TNN.2010.2046179

In the last two decades, many online fault/noise injection algorithms have been developed to attain a fault tolerant neural network. However, not much theoretical works related to their convergence and objective functions have been reported. This paper studies six common fault/noise-injection-based online learning algorithms for radial basis function (RBF) networks, namely 1) injecting additive input noise, 2) injecting additive/multiplicative weight noise, 3) injecting multiplicative node noise, 4) injecting multiweight fault (random disconnection of weights), 5) injecting multinode fault during training, and 6) weight decay with injecting multinode fault. Based on the Gladyshev theorem, we show that the convergence of these six online algorithms is almost sure. Moreover, their true objective functions being minimized are derived. For injecting additive input noise during training, the objective function is identical to that of the Tikhonov regularizer approach. For injecting additive/multiplicative weight noise during training, the objective function is the simple mean square training error. Thus, injecting additive/multiplicative weight noise during training cannot improve the fault tolerance of an RBF network. Similar to injective additive input noise, the objective functions of other fault/noise-injection-based online algorithms contain a mean square error term and a specialized regularization term.