Interactive, Cloud-Native Workflows on HPC Using KNoC

• Published in: High Performance Computing. ISC High Performance 2022 International Workshops pp. 221 - 232
• Main Authors: Maliaroudakis, Evangelos, Chazapis, Antony, Kanterakis, Alexandros, Marazakis, Manolis, Bilas, Angelos
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Cloud-HPC convergence; Kubernetes extensions; Reproducible workflows; Virtual kubelet
• ISBN: 9783031232190; 3031232194
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-031-23220-6_15

Cloud and HPC platforms differentiate by many aspects, but both can run applications in identical contexts using containers. In this paper we present KNoC, an open-source virtual node (kubelet) for Kubernetes that transparently manages the container lifecycle on a remote HPC cluster using Slurm and Singularity. Our goal is on one hand to allow HPC users to leverage existing cloud-native tools, such as the popular Argo Workflows language to express complex data-processing pipelines, while on the other hand enabling Cloud setups to exploit computing resources available in HPC centers. KNoC bridges Cloud and HPC, transforming Argo to a cross-environment, portable solution, which allows the combination of Cloud-based tools and HPC steps into the same workflow, controlled and monitored through an interactive frontend. Deploying KNoC requires only a secure shell connection to the cluster’s login node. We describe the design and implementation of KNoC, and evaluate the integration using several proof-of-concept workflows.