Block-Wise Dynamic-Precision Neural Network Training Acceleration via Online Quantization Sensitivity Analytics

Data quantization is an effective method to accelerate neural network training and reduce power consumption. However, it is challenging to perform low-bit quantized training: the conventional equal-precision quantization will lead to either high accuracy loss or limited bit-width reduction, while ex...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Liu, Ruoyang, Chenhan Wei, Yang, Yixiong, Wang, Wenxun, Yang, Huazhong, Liu, Yongpan
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 31.10.2022
Subjects
Online AccessGet full text

Cover

Loading…
More Information
Summary:Data quantization is an effective method to accelerate neural network training and reduce power consumption. However, it is challenging to perform low-bit quantized training: the conventional equal-precision quantization will lead to either high accuracy loss or limited bit-width reduction, while existing mixed-precision methods offer high compression potential but failed to perform accurate and efficient bit-width assignment. In this work, we propose DYNASTY, a block-wise dynamic-precision neural network training framework. DYNASTY provides accurate data sensitivity information through fast online analytics, and maintains stable training convergence with an adaptive bit-width map generator. Network training experiments on CIFAR-100 and ImageNet dataset are carried out, and compared to 8-bit quantization baseline, DYNASTY brings up to \(5.1\times\) speedup and \(4.7\times\) energy consumption reduction with no accuracy drop and negligible hardware overhead.
ISSN:2331-8422