DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

• Published in: 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID) pp. 172 - 181
• Main Authors: Nicolae, Bogdan, Li, Jiali, Wozniak, Justin M., Bosilca, George, Dorier, Matthieu, Cappello, Franck
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.05.2020
• Subjects: Checkpointing; Computational modeling; Data models; deep learning; fine-grain asynchronous I/O; Machine learning; multi-level data persistence; Neural networks; Scalability; Training
• DOI: 10.1109/CCGrid49817.2020.00-76

In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.