Weight asynchronous update: Improving the diversity of filters in a deep convolutional network

• Published in: Computational visual media (Beijing) Vol. 6; no. 4; pp. 455 - 466
• Main Authors: Zhang, Dejun, He, Linchao, Luo, Mengting, Xu, Zhanya, He, Fazhi
• Format: Journal Article
• Language: English
• Published: Beijing Tsinghua University Press 01.12.2020; Springer Nature B.V; SpringerOpen
• Subjects: Artificial Intelligence; Computer Graphics; Computer Science; convolutional filter; deep convolutional network; Image Processing and Computer Vision; model compression; Network topologies; Neural networks; Research Article; Training; User Interfaces and Human Computer Interaction; Visual tasks; Weight; deep convolutional network; model compression; convolutional filter
• ISSN: 2096-0433; 2096-0662
• DOI: 10.1007/s41095-020-0185-5

Deep convolutional networks have obtained remarkable achievements on various visual tasks due to their strong ability to learn a variety of features. A well-trained deep convolutional network can be compressed to 20%–40% of its original size by removing filters that make little contribution, as many overlapping features are generated by redundant filters. Model compression can reduce the number of unnecessary filters but does not take advantage of redundant filters since the training phase is not affected. Modern networks with residual, dense connections and inception blocks are considered to be able to mitigate the overlap in convolutional filters, but do not necessarily overcome the issue. To do so, we propose a new training strategy, weight asynchronous update, which helps to significantly increase the diversity of filters and enhance the representation ability of the network. The proposed method can be widely applied to different convolutional networks without changing the network topology. Our experiments show that the stochastic subset of filters updated in different iterations can significantly reduce filter overlap in convolutional networks. Extensive experiments show that our method yields noteworthy improvements in neural network performance.