Which Scaling Rule Applies to Artificial Neural Networks

• Published in: Advances in Artificial Intelligence and Applied Cognitive Computing pp. 381 - 407
• Main Author: Végh, János
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing 2021
• Series: Transactions on Computational Science and Computational Intelligence
• Subjects: Artificial intelligence; Computing efficiency; Energy efficiency; Neural network; Scaling rule; Temporal logic
• ISBN: 9783030702953; 3030702952
• ISSN: 2569-7072; 2569-7080
• DOI: 10.1007/978-3-030-70296-0_30

Although an Artificial Neural Network (ANN) is a biology-mimicking system, it is built from components designed/fabricated for use in conventional computing, and it is created by experts trained in conventional computing; all of them are using the classic computing paradigm. As von Neumann in his classic “First Draft” warned, because the data transfer time is neglected in the model he used, using a “too fast processor” vitiates the procedure; furthermore, that using his paradigm for imitating neuronal operations is unsound. This means that at least doubly unsound to apply his paradigm to describe scaling ANNs. The common experience shows that making actively cooperating and communicating computing systems, using segregated single processors, has severe performance limitations; and that fact cannot be explained using his classic paradigm. The achievable payload computing performance of those systems sensitively depends on their workload type, and this effect is only poorly known. The type of the workload that the Artificial Intelligence (AI)-based systems generate leads to an exceptionally low payload computational performance. Unfortunately, the initial successes of demo systems that comprise only a few “neurons” and solve simple tasks are misleading: the scaling of processor-based ANN systems is strongly non-linear. The chapter discusses some major limiting factors that affect their performance. It points out that for building biology-mimicking large systems, it is inevitable to perform drastic changes in the present computing paradigm; namely, instead of neglecting the transfer time, a proper method to consider it shall be developed. The temporal behavior enables us to comprehend the technical implementation of computing components and architectures.