HMC: Hybrid model compression method based on layer sensitivity grouping

• Published in: PloS one Vol. 18; no. 10; p. e0292517
• Main Authors: Yang, Guoliang, Yu, Shuaiying, Yang, Hao, Nie, Ziling, Wang, Jixiang
• Format: Journal Article
• Language: English
• Published: San Francisco Public Library of Science 09.10.2023; Public Library of Science (PLoS)
• Subjects: Accuracy; Analysis; Biology and Life Sciences; Compression; Compression ratio; Computer and Information Sciences; Data compression; Deep learning; Information management; Machine learning; Methods; Modelling; Neural networks; Physical Sciences; Redundancy; Research and Analysis Methods; Sensitivity; Tensors; Voice recognition; China
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0292517

Previous studies have shown that deep models are often over-parameterized, and this parameter redundancy makes deep compression possible. The redundancy of model weight is often manifested as low rank and sparsity. Ignoring any part of the two or the different distributions of these two characteristics in the model will lead to low accuracy and a low compression rate of deep compression. To make full use of the difference between low-rank and sparsity, a unified framework combining low-rank tensor decomposition and structured pruning is proposed: a hybrid model compression method based on sensitivity grouping (HMC). This framework unifies the existing additive hybrid compression method (AHC) and the non-additive hybrid compression method (NaHC) proposed by us into one model. The latter group the network according to the sensitivity difference of the convolutional layer to different compression methods, which can better integrate the low rank and sparsity of the model compared with the former. Experiments show that our approach achieves a better trade-off between test accuracy and compression ratio when compressing the ResNet family of models than other recent compression methods using a single strategy or additive hybrid compression.