SPMgr: Dynamic workflow manager for sampling and filtering data streams over Apache Storm

In this article, we address dynamic workflow management for sampling and filtering data streams in Apache Storm. As many sensors generate data streams continuously, we often use sampling to choose some representative data or filtering to remove unnecessary data. Apache Storm is a real-time distribut...

Full description

Saved in:
Bibliographic Details
Published inInternational journal of distributed sensor networks Vol. 15; no. 7; p. 155014771986220
Main Authors Kim, Youngkuk, Son, Siwoon, Moon, Yang-Sae
Format Journal Article
LanguageEnglish
Published London, England SAGE Publications 01.07.2019
Hindawi - SAGE Publishing
Subjects
data filtering
distributed processing
Apache Storm
Data stream
workflow management
data sampling
Online AccessGet full text

Cover

More Information
Summary:In this article, we address dynamic workflow management for sampling and filtering data streams in Apache Storm. As many sensors generate data streams continuously, we often use sampling to choose some representative data or filtering to remove unnecessary data. Apache Storm is a real-time distributed processing platform suitable for handling large data streams. Storm, however, must stop the entire work when it changes the input data structure or processing algorithm as it needs to modify, redistribute, and restart the programs. In addition, for effective data processing, we often use Storm with Kafka and databases, but it is difficult to use these platforms in an integrated manner. In this article, we derive the problems when applying sampling and filtering algorithms to Storm and propose a dynamic workflow management model that solves these problems. First, we present the concept of a plan consisting of input, processing, and output modules of a data stream. Second, we propose Storm Plan Manager, which can operate Storm, Kafka, and database as a single integrated system. Storm Plan Manager is an integrated workflow manager that dynamically controls sampling and filtering of data streams through plans. Third, as a key feature, Storm Plan Manager provides a Web client interface to visually create, execute, and monitor plans. In this article, we show the usefulness of the proposed Storm Plan Manager by presenting its design, implementation, and experimental results in order.
ISSN:1550-1329
1550-1477
1550-1477
DOI:10.1177/1550147719862206