A Hierarchical, Bulk-Synchronous Stochastic Gradient Descent Algorithm for Deep-Learning Applications on GPU Clusters

The training data and models are becoming increasingly large in many deep-learning applications. Large-scale distributed processing is employed to accelerate training. Increasing the number of learners in synchronous and asynchronous stochastic gradient descent presents challenges to convergence and...

Full description

Saved in:
Bibliographic Details
Published in2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) pp. 818 - 821
Main Authors Guojing Cong, Bhardwaj, Onkar
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.12.2017
Subjects
Aggregates
Bandwidth
Convergence
Deep learning
distributed algorithm
GPU
Graphics processing units
Servers
Stochastic gradient descent
Stochastic processes
Training
Online AccessGet full text

Cover

More Information
Summary:The training data and models are becoming increasingly large in many deep-learning applications. Large-scale distributed processing is employed to accelerate training. Increasing the number of learners in synchronous and asynchronous stochastic gradient descent presents challenges to convergence and communication performance. We present our hierarchical, bulk-synchronous stochastic gradient algorithm that effectively balances execution time and accuracy for training in deep-learning applications on GPU clusters. It achieves much better convergence and execution time at scale in comparison to asynchronous stochastic gradient descent implementations. When deployed on a cluster of 128 GPUs, our implementation achieves up to 56 times speedups over the sequential stochastic gradient descent with similar test accuracy for our target application.
DOI:10.1109/ICMLA.2017.00-56