Random Sketching for Neural Networks With ReLU

• Published in: IEEE transaction on neural networks and learning systems Vol. 32; no. 2; pp. 748 - 762
• Main Authors: Wang, Di, Zeng, Jinshan, Lin, Shao-Bo
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Approximation; Back propagation; Back propagation networks; Computer applications; Convergence; Deep learning; Design optimization; Generalization error; Humans; Kernel; Learning algorithms; Least squares method; Least-Squares Analysis; Linear Models; Machine Learning; Mathematical analysis; Neural networks; Neural Networks, Computer; Optimization; random sketching; rectified linear unit (ReLU); Theoretical analysis; Training

Training neural networks is recently a hot topic in machine learning due to its great success in many applications. Since the neural networks' training usually involves a highly nonconvex optimization problem, it is difficult to design optimization algorithms with perfect convergence guarantees...