Towards Compact 1-bit CNNs via Bayesian Learning

• Published in: International journal of computer vision Vol. 130; no. 2; pp. 201 - 225
• Main Authors: Zhao, Junhe, Xu, Sheng, Zhang, Baochang, Gu, Jiaxin, Doermann, David, Guo, Guodong
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2022; Springer; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Artificial neural networks; Back propagation; Bayesian analysis; Comparative analysis; Computer Imaging; Computer Science; Computer vision; Data mining; Deep learning; Electronic devices; Image Processing and Computer Vision; Kernels; Machine learning; Machine vision; Neural networks; Normal distribution; Object recognition; Pattern Recognition; Pattern Recognition and Graphics; Performance enhancement; Smartphones; Vision; Quantization; Pruning; Binarized network; Bayesian learning
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-021-01543-y

Deep convolutional neural networks (DCNNs) have dominated as the best performers on almost all computer vision tasks over the past several years. However, it remains a major challenge to deploy these powerful DCNNs in resource-limited environments, such as embedded devices and smartphones. To this end, 1-bit CNNs have emerged as a feasible solution as they are much more resource-efficient. Unfortunately, they often suffer from a significant performance drop compared to their full-precision counterparts. In this paper, we propose a novel Bayesian Optimized compact 1-bit CNNs (BONNs) model, which has the advantage of Bayesian learning, to improve the performance of 1-bit CNNs significantly. BONNs incorporate the prior distributions of full-precision kernels, features, and filters into a Bayesian framework to construct 1-bit CNNs in a comprehensive end-to-end manner. The proposed Bayesian learning algorithms are well-founded and used to optimize the network simultaneously in different kernels, features, and filters, which largely improves the compactness and capacity of 1-bit CNNs. We further introduce a new Bayesian learning-based pruning method for 1-bit CNNs, which significantly increases the model efficiency with very competitive performance. This enables our method to be used in a variety of practical scenarios. Extensive experiments on the ImageNet, CIFAR, and LFW datasets show that BONNs achieve the best in classification performance compared to a variety of state-of-the-art 1-bit CNN models. In particular, BONN achieves a strong generalization performance on the object detection task.