Differential Evolution Based Layer-Wise Weight Pruning for Compressing Deep Neural Networks

• Published in: Sensors (Basel, Switzerland) Vol. 21; no. 3; p. 880
• Main Authors: Wu, Tao, Li, Xiaoyang, Zhou, Deyun, Li, Na, Shi, Jiao
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI 28.01.2021; MDPI AG
• Subjects: differential evolution; neural network compression; sparse network; weight pruning; differential evolution; weight pruning; neural network compression; sparse network
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s21030880

Deep neural networks have evolved significantly in the past decades and are now able to achieve better progression of sensor data. Nonetheless, most of the deep models verify the ruling maxim in deep learning-bigger is better-so they have very complex structures. As the models become more complex, the computational complexity and resource consumption of these deep models are increasing significantly, making them difficult to perform on resource-limited platforms, such as sensor platforms. In this paper, we observe that different layers often have different pruning requirements, and propose a differential evolutionary layer-wise weight pruning method. Firstly, the pruning sensitivity of each layer is analyzed, and then the network is compressed by iterating the weight pruning process. Unlike some other methods that deal with pruning ratio by greedy ways or statistical analysis, we establish an optimization model to find the optimal pruning sensitivity set for each layer. Differential evolution is an effective method based on population optimization which can be used to address this task. Furthermore, we adopt a strategy to recovery some of the removed connections to increase the capacity of the pruned model during the fine-tuning phase. The effectiveness of our method has been demonstrated in experimental studies. Our method compresses the number of weight parameters in LeNet-300-100, LeNet-5, AlexNet and VGG16 by 24×, 14×, 29× and 12×, respectively.