Distributed Training for High Resolution Images: A Domain and Spatial Decomposition Approach

In this work we developed two Pytorch libraries using the PyTorch RPC interface for distributed deep learning approaches on high resolution images. The spatial decomposition library allows for distributed training on very large images, which otherwise wouldn't be possible on a single GPU. The d...

Full description

Saved in:
Bibliographic Details
Published in2021 IEEE/ACM Redefining Scalability for Diversely Heterogeneous Architectures Workshop (RSDHA) pp. 27 - 33
Main Authors Tsaris, Aristeidis, Hinkle, Jacob, Lunga, Dalton, Dias, Philipe Ambrozio
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.11.2021
Subjects
distributed deep learning
Graphics processing units
high resolution images
Parallel processing
Pipelines
scalability
Solid modeling
Three-dimensional displays
Throughput
Training
Online AccessGet full text

Cover

More Information
Summary:In this work we developed two Pytorch libraries using the PyTorch RPC interface for distributed deep learning approaches on high resolution images. The spatial decomposition library allows for distributed training on very large images, which otherwise wouldn't be possible on a single GPU. The domain parallelism library allows for distributed training across multiple domain unlabeled data, by leveraging the domain separation architecture. Both of those libraries where tested on the Summit supercomputer at Oak Ridge National Laboratory at a moderate scale.
DOI:10.1109/RSDHA54838.2021.00009