A Selective Modular Neural Network Framework

• Published in: 2019 IEEE 10th International Conference on Awareness Science and Technology (iCAST) pp. 1 - 6
• Main Authors: Intisar, Chowdhury Md, Zhao, Qiangfu
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.10.2019
• Subjects: Artificial neural networks; Biological neural networks; Computational modeling; deep learning; distillation; ensemble learning; Knowledge engineering; model compression; Modular Neural Network; Routing; Training
• ISSN: 2325-5994
• DOI: 10.1109/ICAwST.2019.8923334

We propose a simple yet effective modular neural network framework for multi-class classification. The proposed framework significantly reduces the number of parameters while maintaining the accuracy comparable to more complex deep neural networks such as AlexNet and ResNet. The framework primarily consists of two major parts, namely a routing module and a set of expert modules. Each of the expert module is a binary classifier trained in Round Robin fashion. The novelty in this literature is on how we leverage the routing module to select only a small set of expert modules for each input datum during the testing phase. The selection of the expert module is carried out based on the routing module's soft-max scores for top-2 classes. In addition, each of the modules in the framework is very minimalist by design. The framework was evaluated on three real-world public data-sets. Empirical results show that with only a budget of 0.35M parameters we can achieve 99.56% accuracy on MNIST (0.2% improvement over AlexNet). For the Fashion-Mnist dataset, with the same budget and network architecture as MNIST we achieved 91.00% accuracy. For the UCI-HAR signal data, with a budget of only 2.5M, we can achieve an accuracy of 96.00% which is comparable to AlexNet (96.30%).