Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning

• Main Authors: Koloskova, Anastasia, Stich, Sebastian U, Jaggi, Martin
• Format: Journal Article
• Language: English
• Published: 16.06.2022
• Subjects: Computer Science - Distributed, Parallel, and Cluster Computing; Computer Science - Learning; Mathematics - Optimization and Control

We study the asynchronous stochastic gradient descent algorithm for distributed training over $n$ workers which have varying computation and communication frequency over time. In this algorithm, workers compute stochastic gradients in parallel at their own pace and return those to the server without...