Triggerflow: Trigger-based orchestration of serverless workflows

As more applications are being moved to the Cloud thanks to serverless computing, it is increasingly necessary to support the native life cycle execution of those applications in the data center. But existing cloud orchestration systems either focus on short-running workflows (like IBM Composer or A...

Full description

Saved in:
Bibliographic Details
Published inFuture generation computer systems Vol. 124; pp. 215 - 229
Main Authors Arjona, Aitor, López, Pedro García, Sampé, Josep, Slominski, Aleksander, Villard, Lionel
Format Journal Article
LanguageEnglish
Published Elsevier B.V 01.11.2021
Subjects
Event-based
Orchestration
Serverless
Orchestration
Serverless
Event-based
Online AccessGet full text

Cover

More Information
Summary:As more applications are being moved to the Cloud thanks to serverless computing, it is increasingly necessary to support the native life cycle execution of those applications in the data center. But existing cloud orchestration systems either focus on short-running workflows (like IBM Composer or Amazon Step Functions Express Workflows) or impose considerable overheads for synchronizing massively parallel jobs (Azure Durable Functions, Amazon Step Functions). None of them are open systems enabling extensible interception and optimization of custom workflows. We present Triggerflow: an extensible Trigger-based Orchestration architecture for serverless workflows. We demonstrate that Triggerflow is a novel serverless building block capable of constructing different reactive orchestrators (State Machines, Directed Acyclic Graphs, Workflow as code, Federated Learning orchestrator). We also validate that it can support high-volume event processing workloads, auto-scale on demand with scale down to zero when not used, and transparently guarantee fault tolerance and efficient resource usage when orchestrating long running scientific workflows. •Analysis of current serverless orchestration systems for scientific workflows.•Specification of a novel trigger-based architecture for serverless orchestration.•Demonstration of how Triggerflow orchestrates DAGs, state machines, workflow as code.•Prototype implementation of the serverless auto-scalable architecture on Kubernetes.•Evaluation of serverless orchestration performance, fault tolerance and efficiency.
ISSN:0167-739X
1872-7115
DOI:10.1016/j.future.2021.06.004