Khaos: Dynamically Optimizing Checkpointing for Dependable Distributed Stream Processing

Distributed Stream Processing systems are becoming an increasingly essential part of Big Data processing platforms as users grow ever more reliant on their ability to provide fast access to new results. As such, making timely decisions based on these results is dependent on a system's ability t...

Full description

Saved in:
Bibliographic Details
Published in2022 17th Conference on Computer Science and Intelligence Systems (FedCSIS) Vol. 30; pp. 553 - 561
Main Authors Geldenhuys, Morgan K., Pfister, Benjamin J. J., Scheinert, Dominik, Thamsen, Lauritz, Kao, Odej
Format Conference Proceeding Journal Article
LanguageEnglish
Published Polish Information Processing Society 01.01.2022
Subjects
Chaos
Chaos Engineering
Cloud
Distributed Stream Processing
Fault tolerance
Fault tolerant systems
Parallel processing
Parallel Profiling
Probability
QoS Modeling
Quality of service
Runtime
Runtime Optimization
Online AccessGet full text
ISSN2300-5963
DOI10.15439/2022F225

Cover

More Information
Summary:Distributed Stream Processing systems are becoming an increasingly essential part of Big Data processing platforms as users grow ever more reliant on their ability to provide fast access to new results. As such, making timely decisions based on these results is dependent on a system's ability to tolerate failure. Typically, these systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing checkpoint and rollback recovery. However, owing to the statistical probability of partial failures occurring in these distributed environments and the variability of workloads upon which jobs are expected to operate, static configurations will often not meet Quality of Service constraints with low overhead.In this paper we present Khaos, a new approach which utilizes the parallel processing capabilities of cloud orchestration technologies for the automatic runtime optimization of fault tolerance configurations in Distributed Stream Processing jobs. Our approach employs three subsequent phases which borrows from the principles of Chaos Engineering: establish the steady-state processing conditions, conduct experiments to better understand how the system performs under failure, and use this knowledge to continuously minimize Quality of Service violations. We implemented Khaos prototypically together with Apache Flink and demonstrate its usefulness experimentally.
ISSN:2300-5963
DOI:10.15439/2022F225