KubeAdaptor: A docking framework for workflow containerization on Kubernetes

• Published in: Future generation computer systems Vol. 148; pp. 584 - 599
• Main Authors: Shan, Chenggang, Xia, Yuanqing, Zhan, Yufeng, Zhang, Jinhui
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.11.2023
• Subjects: Cloud-native; Containerization; Event trigger mechanism; Kubernetes; Task scheduling; Workflow systems; Task scheduling; Containerization; Workflow systems; Event trigger mechanism; Cloud-native; Kubernetes
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2023.06.022

As Kubernetes becomes the infrastructure of the cloud-native era, the integration of workflow systems with Kubernetes is gaining more and more popularity. However, when migrating the workflow systems to the Kubernetes platform, the inconsistency between the workflow scheduling algorithms and Kubernetes in task scheduling order seriously reduces the execution efficiency of the workflow systems. Besides, the integration of existing container-based workflow systems with Kubernetes lacks a general docking framework and requires various built-in tools and specialized technique supports, which brings high research costs and inconvenience to users. To this end, this paper proposes a cloud-native workflow engine named KubeAdaptor, a general docking framework to integrate workflow systems with Kubernetes and implement workflow containerization on Kubernetes, aiming to ensure a consistent task scheduling order between workflow scheduling algorithms in workflow systems and Kubernetes. We present the KubeAdaptor architecture and elaborate on the functionality implementation, fault tolerance management, and the event trigger mechanism within the KubeAdaptor. Experimental results on four real-world scientific workflows show that the KubeAdaptor as a docking framework ensures the consistency of the workflow scheduling algorithms and Kubernetes in the task scheduling order. Compared with the state-of-the-art Argo workflow engine, the KubeAdaptor achieves better performance in terms of the average execution time of the task pod, average workflow lifecycle, and resource usage rate. •Design a cloud-native workflow engine that works as a docking framework to integrate workflow systems with Kubernetes.•A flexible workflow definition method is proposed to customize workflows and serves as an interface to accommodate task execution sequences optimized by workflow scheduling algorithms.•Implement a workflow injection module, the resource gathering module for experimental evaluation, and an event trigger mechanism.•Provide a containerized solution with resource loads for workflow tasks to run four real-world workflow applications.