Information Bottleneck Classification in Extremely Distributed Systems

• Published in: Entropy (Basel, Switzerland) Vol. 22; no. 11; p. 1237
• Main Authors: Ullmann, Denis, Rezaeifar, Shideh, Taran, Olga, Holotyak, Taras, Panos, Brandon, Voloshynovskiy, Slava
• Format: Journal Article
• Language: English
• Published: Switzerland MDPI 30.10.2020; MDPI AG
• Subjects: classification; decentralized model; deep networks; information bottleneck principle; rate-distortion theory; information bottleneck principle; decentralized model; deep networks; rate-distortion theory; classification
• ISSN: 1099-4300; 1099-4300
• DOI: 10.3390/e22111237

We present a new decentralized classification system based on a distributed architecture. This system consists of distributed nodes, each possessing their own datasets and computing modules, along with a centralized server, which provides probes to classification and aggregates the responses of nodes for a final decision. Each node, with access to its own training dataset of a given class, is trained based on an auto-encoder system consisting of a fixed a pre-trained and a Hence, these auto-encoders are highly dependent on the class probability distribution for which the reconstruction distortion is minimized. Alternatively, when an encoding-quantizing-decoding node observes data from different distributions, unseen at training, there is a mismatch, and such a decoding is not optimal, leading to a significant increase of the reconstruction distortion. The final classification is performed at the centralized classifier that votes for the class with the minimum reconstruction distortion. In addition to the system applicability for applications facing big-data communication problems and or requiring private classification, the above distributed scheme creates a theoretical bridge to the information bottleneck principle. The proposed system demonstrates a very promising performance on basic datasets such as MNIST and FasionMNIST.