NEPTUNE: Real Time Stream Processing for Internet of Things and Sensing Environments

• Published in: 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) pp. 1143 - 1152
• Main Authors: Buddhika, Thilina, Pallickara, Shrideep
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.05.2016
• Subjects: Bandwidth; Distributed stream processing; High-throughput data processing; Internet of things; Message systems; Program processors; Real-time systems; Sensors; Throughput
• ISSN: 1530-2075
• DOI: 10.1109/IPDPS.2016.43

Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in accomplishing this include: high data arrival rates, buffer overflows, context-switches, and object creation overheads. We propose a holistic framework that addresses the CPU, memory, network, and kernel issues involved in stream processing. Our prototype, Neptune, builds on our Granules cloud runtime. The framework maximizes bandwidth utilization in the presence of small messages via the use of buffering and dynamic compactions of packets based on payload entropy. Our use of thread-pools and batched processing reduces context switches and improves effective CPU utilizations. NEPTUNE alleviates memory pressure that can lead to swapping, page faults, and thrashing through efficient reuse of objects. To cope with buffer overflows we rely on flow control and throttling the preceding stages of a processing pipeline. Our benchmarks demonstrate the suitability of the Neptune and we contrast our performance with Apache Storm, the dominant stream-processing framework developed by Twitter. At a single node, we are able to achieve a processing rate of ~2 million stream packets per-second. In a distributed setup, we achieved a rate of ~100 million packets per-second.