Elastic Allocator: An Adaptive Task Scheduler for Streaming Query in the Cloud

Many big data applications receive and process data in real time. These data, also known as data streams, are generated continuously and processed online in a low latency manner. Data stream is prone to change dramatically in volume, since its workload may have a variation of several orders between...

Full description

Saved in:
Bibliographic Details
Published in2014 IEEE 8th International Symposium on Service Oriented System Engineering pp. 284 - 289
Main Authors Zheng Han, Rui Chu, Haibo Mi, Huaimin Wang
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2014
Subjects
Bandwidth
Cloud computing
Clustering algorithms
Computer architecture
data stream
elastic
Linear programming
Resource management
Storms
stream query
task allocation
Online AccessGet full text

Cover

More Information
Summary:Many big data applications receive and process data in real time. These data, also known as data streams, are generated continuously and processed online in a low latency manner. Data stream is prone to change dramatically in volume, since its workload may have a variation of several orders between peak and valley periods. Fully provisioning resources for stream processing to handle the peak load is costly, while over-provisioning is wasteful when to deal with lightweight workload. Cloud computing emphasizes that resource should be utilized economically and elastically. An open question is how to allocate query task adaptively to keeping up the input rate of the data stream. Previous work focuses on using either local or global capacity information to improve the cluster CPU resource utilization, while the bandwidth utilization which is also critical to the system throughput is ignored or simplified. In this paper, we formalize the operator placement problem considering both the CPU and bandwidth usage, and introduce the Elastic Allocator. The Elastic Allocator uses a quantitative method to evaluate a node's capacity and bandwidth usage, and exploit both the local and global resource information to allocate the query task in a graceful manner to achieve high resource utilization. The experimental results and a simple prototype built on top of Storm finally demonstrate that Elastic Allocator is adaptive and feasible in cloud computing environment, and has an advantage of improving and balancing system resource utilization.
DOI:10.1109/SOSE.2014.40