ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

Neural network training is inherently sequential where the layers finish the forward propagation in succession, followed by the calculation and back-propagation of gradients (based on a loss function) starting from the last layer. The sequential computations significantly slow down neural network tr...

Full description

Saved in:
Bibliographic Details
Published in2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO) pp. 1092 - 1105
Main Authors Janfaza, Vahid, Mandal, Shantanu, Mahmud, Farabi, Muzahid, Abdullah
Format Conference Proceeding
LanguageEnglish
Published ACM 28.10.2023
Subjects
Adaptation models
Artificial neural networks
Computational modeling
Hardware accelerators
Memory management
Prediction
Predictive models
Systolic arrays
Tensors
Training
Online AccessGet full text

Cover

More Information
Summary:Neural network training is inherently sequential where the layers finish the forward propagation in succession, followed by the calculation and back-propagation of gradients (based on a loss function) starting from the last layer. The sequential computations significantly slow down neural network training, especially the deeper ones. Prediction has been successfully used in many areas of computer architecture to speed up sequential processing. Therefore, we propose ADA-GP, which uses gradient prediction adaptively to speed up deep neural network (DNN) training while maintaining accuracy. ADA-GP works by incorporating a small neural network to predict gradients for different layers of a DNN model. ADA-GP uses a novel tensor reorganization method to make it feasible to predict a large number of gradients. ADA-GP alternates between DNN training using backpropagated gradients and DNN training using predicted gradients. ADA-GP adaptively adjusts when and for how long gradient prediction is used to strike a balance between accuracy and performance. Last but not least, we provide a detailed hardware extension in a typical DNN accelerator to realize the speed up potential from gradient prediction. Our extensive experiments with fifteen DNN models show that ADA-GP can achieve an average speed up of 1.47× with similar or even higher accuracy than the baseline models. Moreover, it consumes, on average, 34% less energy due to reduced off-chip memory accesses compared to the baseline accelerator.CCS CONCEPTS* Computing methodologies → Neural networks; * Computer systems organization → Neural networks.