Parla: A Python Orchestration System for Heterogeneous Architectures

• Published in: SC22: International Conference for High Performance Computing, Networking, Storage and Analysis pp. 1 - 15
• Main Authors: Lee, Hochan, Ruys, William, Henriksen, Ian, Peters, Arthur, Yan, Yineng, Stephens, Sean, You, Bozhi, Fingler, Henrique, Burtscher, Martin, Gligoric, Milos, Schulz, Karl, Pingali, Keshav, Rossbach, Christopher J., Erez, Mattan, Biros, George
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.11.2022
• Subjects: Codes; heterogeneous computing; High performance computing; load balancing and scheduling algorithms; Parallel application frameworks; Parallel programming; Prefetching; Runtime; Scheduling algorithms; task based parallelism; Writing
• ISSN: 2167-4337
• DOI: 10.1109/SC41404.2022.00056

Python's ease of use and rich collection of numeric libraries make it an excellent choice for rapidly developing scientific applications. However, composing these libraries to take advantage of complex heterogeneous nodes is still difficult. To simplify writing multi-device code, we created Parla, a heterogeneous task-based programming framework that fully supports Python's scientific programming stack. Parla's API is based on Python decorators and allows users to wrap code in Parla tasks for parallel execution. Parla arrays enable automatic movement of data between devices. The Parla runtime handles resource-aware mapping, scheduling, and execution of tasks. Compared to other Python tasking systems, Parla is unique in its parallelization of tasks within a single process, its GPU context and resource-aware runtime, and its design around gradual adoption to provide easy migration of and integration into existing Python applications. We show that Parla can achieve performance competitive with hand-optimized code while improving ease of development.