Containerized Workflow Builder for Kubernetes

Docker and Kubernetes have recently dominated the whole cloud-native technology ecosystem and speeded up the containerization process of workflows. By optimizing the task scheduling order, workflow scheduling algorithms signifi-cantly improve the execution efficiency of workflows. However, existing...

Full description

Saved in:
Bibliographic Details
Published in2021 IEEE 23rd Int Conf on High Performance Computing & Communications; 7th Int Conf on Data Science & Systems; 19th Int Conf on Smart City; 7th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys) pp. 685 - 692
Main Authors Shan, Chenggang, Wang, Guan, Xia, Yuanqing, Zhan, Yufeng, Zhang, Jinhui
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.12.2021
Subjects
Containerization
Ecosystems
Event Trigger Mechanism
Kubernetes
Production
Real-time systems
Scheduling
Scheduling algorithms
Smart cities
Task analysis
Workflow Scheduling
Online AccessGet full text

Cover

More Information
Summary:Docker and Kubernetes have recently dominated the whole cloud-native technology ecosystem and speeded up the containerization process of workflows. By optimizing the task scheduling order, workflow scheduling algorithms signifi-cantly improve the execution efficiency of workflows. However, existing works do not support workflow scheduling following the optimized task scheduling order on Kubernetes. How to ensure the consistency of workflow scheduling algorithms and Kubernetes scheduler in task scheduling order is challenging. In this paper, we propose a Containerized Workflow Builder (CWB) for Kubernetes, a framework able to implement con-tainerization of workflows in a two-level scheduling scheme and connect workflow scheduling algorithms to Kubernetes, ensuring the consistency of the task scheduling order. It uses the Informer component to watch the underlying resource events of Kubernetes, providing the event trigger mechanism to respond to event callback in real-time and implement containerization of workflows by the Client-go package. Experimental results show that our proposed CWB ensures the consistency of the workflow scheduling algorithms and Kubernetes scheduler in the task scheduling order. Compared with the state-of-the-art, the CWB achieves better performance in terms of average task pod execution time, average workflow lifecycle, and resource usage rate.
DOI:10.1109/HPCC-DSS-SmartCity-DependSys53884.2021.00113