Spatial Shift Point-Wise Quantization

• Published in: IEEE access Vol. 8; pp. 207683 - 207690
• Main Authors: Kim, Eunhui, Lee, Kyong-Ha
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Artificial neural networks; Compact neural network design; Computational modeling; Convolution; convolution neural network; Data models; Datasets; Degradation; Lightweight; lightweight modeling; Machine learning; Mathematical model; Mathematical models; Measurement; Model accuracy; Neural networks; Parameters; quantization; Quantization (signal); Regeneration; Source code; Training
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2020.3038164

Deep neural networks (DNN) have been applied to numerous artificial-intelligence applications because of their remarkable accuracy. However, computational requirements for deep neural networks are recently skyrocketing far beyond the Moore's Law. In addition to the importance of accuracy, the industry's demand for efficiency in model learning process is increasing. This has led to various attempts to make DNNs more lightweight. Hence, we propose a modeling technique that applies lightweight convolutional neural networks (CNN) to handle the model-learning processes for DNNs. The proposed spatial-shift pointwise quantization (SSPQ) model elegantly combines compact network-design techniques to revitalize DNN quantization efficiency with little accuracy loss. We set the depths of our SSPQ model to 20, 34, and 50 to test against CIFAR10, CIFAR100, and ImageNet datasets, respectively. By applying SSPQ20 to the CIFAR10 dataset, we reduced accuracy degradation by 2.95%, while reducing the number of parameters <inline-formula> <tex-math notation="LaTeX">8.7\times </tex-math></inline-formula>. For the same dataset, our "wide" SSPQ20 variant reduced training parameters <inline-formula> <tex-math notation="LaTeX">1.96\times </tex-math></inline-formula>, compared with the ResNet20 model architecture, which provided a 0.2% improvement. By applying SSPQ34 to the CIFAR100 dataset, we successfully reduced the number of learning parameters <inline-formula> <tex-math notation="LaTeX">8\times </tex-math></inline-formula>, compared with the ResNet34 model with an accuracy degradation of 4.57%. By applying SSPQ50 to the ImageNet dataset, we successfully reduced the number of parameters <inline-formula> <tex-math notation="LaTeX">10.2\times </tex-math></inline-formula> over ResNet50 with an accuracy degradation of only 2.68%. Therefore, using only 9.78MB of learning parameter, the SSPQ50 model guarantees 73.6% accuracy. This is an improved performance by 1.46% in terms of accuracy and 3.62MB in terms of model size compared to the MobileNetV2 lightweight model. The source code is available at https://github.com/Eunhui-Kim/SSPQ .