A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

• Main Authors: Huang, Feiqing, Si, Yuefeng, Zheng, Yao, Li, Guodong
• Format: Journal Article
• Language: English
• Published: 09.12.2021
• Subjects: Computer Science - Learning; Statistics - Machine Learning
• DOI: 10.48550/arxiv.2112.04857

While recently many designs have been proposed to improve the model efficiency of convolutional neural networks (CNNs) on a fixed resource budget, theoretical understanding of these designs is still conspicuously lacking. This paper aims to provide a new framework for answering the question: Is there still any remaining model redundancy in a compressed CNN? We begin by developing a general statistical formulation of CNNs and compressed CNNs via the tensor decomposition, such that the weights across layers can be summarized into a single tensor. Then, through a rigorous sample complexity analysis, we reveal an important discrepancy between the derived sample complexity and the naive parameter counting, which serves as a direct indicator of the model redundancy. Motivated by this finding, we introduce a new model redundancy measure for compressed CNNs, called the $K/R$ ratio, which further allows for nonlinear activations. The usefulness of this new measure is supported by ablation studies on popular block designs and datasets.