Automating Characterization Deployment in Distributed Data Stream Management Systems

• Published in: IEEE transactions on knowledge and data engineering Vol. 29; no. 12; pp. 2669 - 2681
• Main Authors: Wang, Chunkai, Meng, Xiaofeng, Guo, Qi, Weng, Zujian, Yang, Chen
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Automation; Configurations; Data transmission; Delta-sigma modulation; Dynamic scheduling; Feature extraction; incremental learning; Learning systems; Machine learning; Management systems; Media streaming; modeling and prediction; Predictive models; Queries; Query processing; Real-time systems; Reconfiguration; relational query system; Stream processing system; Variations
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2017.2751606

Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level's four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS-Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively.