On the Selection of Weight Decay Parameter for Faulty Networks

• Published in: IEEE transactions on neural networks Vol. 21; no. 8; pp. 1232 - 1244
• Main Authors: Chi Sing Leung, Hong-Jiang Wang, Sum, J
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.08.2010; Institute of Electrical and Electronics Engineers
• Subjects: Algorithms; Animals; Applied sciences; Artificial Intelligence; Computer science; control theory; systems; Computer Simulation - standards; Connectionism. Neural networks; Decay; Electronic commerce; Error analysis; Errors; Exact sciences and technology; Fault tolerance; Faults; Faulty network; Forecasting - methods; generalization error; Humans; Linear Models; Mathematical models; mean prediction error; Networks; Neural networks; Neural Networks (Computer); Nonlinear Dynamics; Permissible error; Predictive Value of Tests; regularization; Testing; weight decay; Radial basis function; Multiplicative noise; mean prediction error; Error estimation; Faulty network; Mean error; Neural network; Generalization; Modeling; Regularization; generalization error; weight decay
• ISSN: 1045-9227; 1941-0093
• DOI: 10.1109/TNN.2010.2049580

The weight-decay technique is an effective approach to handle overfitting and weight fault. For fault-free networks, without an appropriate value of decay parameter, the trained network is either overfitted or underfitted. However, many existing results on the selection of decay parameter focus on fault-free networks only. It is well known that the weight-decay method can also suppress the effect of weight fault. For the faulty case, using a test set to select the decay parameter is not practice because there are huge number of possible faulty networks for a trained network. This paper develops two mean prediction error (MPE) formulae for predicting the performance of faulty radial basis function (RBF) networks. Two fault models, multiplicative weight noise and open weight fault, are considered. Our MPE formulae involve the training error and trained weights only. Besides, in our method, we do not need to generate a huge number of faulty networks to measure the test error for the fault situation. The MPE formulae allow us to select appropriate values of decay parameter for faulty networks. Our experiments showed that, although there are small differences between the true test errors (from the test set) and the MPE values, the MPE formulae can accurately locate the appropriate value of the decay parameter for minimizing the true test error of faulty networks.