Fast Conditional Network Compression Using Bayesian HyperNetworks

• Published in: Machine Learning and Knowledge Discovery in Databases. Research Track Vol. 12977; pp. 330 - 345
• Main Authors: Nguyen, Phuoc, Tran, Truyen, Le, Ky, Gupta, Sunil, Rana, Santu, Nguyen, Dang, Nguyen, Trong, Ryan, Shannon, Venkatesh, Svetha
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Bayesian compression; Bayesian deep learning; Hyper networks; Meta-learning; Network compression
• ISBN: 3030865223; 9783030865221
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-86523-8_20

We introduce a conditional compression problem and propose a fast framework for tackling it. The problem is how to quickly compress a pretrained large neural network into optimal smaller networks given target contexts, e.g., a context involving only a subset of classes or a context where only limited compute resource is available. To solve this, we propose an efficient Bayesian framework to compress a given large network into much smaller size tailored to meet each contextual requirement. We employ a hypernetwork to parameterize the posterior distribution of weights given conditional inputs and minimize a variational objective of this Bayesian neural network. To further reduce the network sizes, we propose a new input-output group sparsity factorization of weights to encourage more sparseness in the generated weights. Our methods can quickly generate compressed networks with significantly smaller sizes than baseline methods.