Efficient and Robust Distributed Matrix Computations via Convolutional Coding

• Published in: IEEE transactions on information theory Vol. 67; no. 9; pp. 6266 - 6282
• Main Authors: Das, Anindya Bijoy, Ramamoorthy, Aditya, Vaswani, Namrata
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cloud computing; Codes; Coding; Complexity theory; Convolutional codes; convolutional coding; Decoding; Distributed computing; Encoding; Lower bounds; Mathematical analysis; Matrix decomposition; Numerical stability; Resilience; Robustness (mathematics); Roundoff error; Storage capacity; straggler; Toeplitz matrix; Upper bounds; Vandermonde matrix
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2021.3095909

Distributed matrix computations - matrix-matrix or matrix-vector multiplications - are well-recognized to suffer from the problem of stragglers (slow or failed worker nodes). Much of prior work in this area is (i) either sub-optimal in terms of its straggler resilience, or (ii) suffers from numerical problems, i.e., there is a blow-up of round-off errors in the decoded result owing to the high condition numbers of the corresponding decoding matrices. Our work presents a convolutional coding approach to this problem that removes these limitations. It is optimal in terms of its straggler resilience, and has excellent numerical robustness as long as the workers' storage capacity is slightly higher than the fundamental lower bound. Moreover, it can be decoded using a fast peeling decoder that only involves add/subtract operations. Our second approach has marginally higher decoding complexity than the first one, but allows us to operate arbitrarily close to the storage capacity lower bound. Its numerical robustness can be theoretically quantified by deriving a computable upper bound on the worst case condition number over all possible decoding matrices by drawing connections with the properties of large block Toeplitz matrices. All above claims are backed up by extensive experiments done on the AWS cloud platform.