Structured Matrices and Their Application in Neural Networks: A Survey

• Published in: New generation computing Vol. 41; no. 3; pp. 697 - 722
• Main Authors: Kissel, Matthias, Diepold, Klaus
• Format: Journal Article
• Language: English
• Published: Tokyo Springer Japan 01.09.2023; Springer Nature B.V
• Subjects: Artificial Intelligence; Benchmarks; Computer architecture; Computer Hardware; Computer Science; Computer Systems Organization and Communication Networks; Neural networks; Software Engineering/Programming and Operating Systems; Sparse matrices; Structured matrices; Matrix structures; Fast inference; Neural network; Efficient propagation
• ISSN: 0288-3635; 1882-7055
• DOI: 10.1007/s00354-023-00226-1

Modern neural network architectures are becoming larger and deeper, with increasing computational resources needed for training and inference. One approach toward handling this increased resource consumption is to use structured weight matrices. By exploiting structures in weight matrices, the computational complexity for propagating information through the network can be reduced. However, choosing the right structure is not trivial, especially since there are many different matrix structures and structure classes. In this paper, we give an overview over the four main matrix structure classes, namely semiseparable matrices, matrices of low displacement rank, hierarchical matrices and products of sparse matrices. We recapitulate the definitions of each structure class, present special structure subclasses, and provide references to research papers in which the structures are used in the domain of neural networks. We present two benchmarks comparing the classes. First, we benchmark the error for approximating different test matrices. Second, we compare the prediction performance of neural networks in which the weight matrix of the last layer is replaced by structured matrices. After presenting the benchmark results, we discuss open research questions related to the use of structured matrices in neural networks and highlight future research directions.