NEPTUNE: Real Time Stream Processing for Internet of Things and Sensing Environments

Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in acco...

Full description

Saved in:
Bibliographic Details
Published in2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) pp. 1143 - 1152
Main Authors Buddhika, Thilina, Pallickara, Shrideep
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.05.2016
Subjects
Online AccessGet full text

Cover

Loading…
Abstract Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in accomplishing this include: high data arrival rates, buffer overflows, context-switches, and object creation overheads. We propose a holistic framework that addresses the CPU, memory, network, and kernel issues involved in stream processing. Our prototype, Neptune, builds on our Granules cloud runtime. The framework maximizes bandwidth utilization in the presence of small messages via the use of buffering and dynamic compactions of packets based on payload entropy. Our use of thread-pools and batched processing reduces context switches and improves effective CPU utilizations. NEPTUNE alleviates memory pressure that can lead to swapping, page faults, and thrashing through efficient reuse of objects. To cope with buffer overflows we rely on flow control and throttling the preceding stages of a processing pipeline. Our benchmarks demonstrate the suitability of the Neptune and we contrast our performance with Apache Storm, the dominant stream-processing framework developed by Twitter. At a single node, we are able to achieve a processing rate of ~2 million stream packets per-second. In a distributed setup, we achieved a rate of ~100 million packets per-second.
AbstractList Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing environments. Data streams generated in such settings must keep pace with generation rates and be processed in real time. Challenges in accomplishing this include: high data arrival rates, buffer overflows, context-switches, and object creation overheads. We propose a holistic framework that addresses the CPU, memory, network, and kernel issues involved in stream processing. Our prototype, Neptune, builds on our Granules cloud runtime. The framework maximizes bandwidth utilization in the presence of small messages via the use of buffering and dynamic compactions of packets based on payload entropy. Our use of thread-pools and batched processing reduces context switches and improves effective CPU utilizations. NEPTUNE alleviates memory pressure that can lead to swapping, page faults, and thrashing through efficient reuse of objects. To cope with buffer overflows we rely on flow control and throttling the preceding stages of a processing pipeline. Our benchmarks demonstrate the suitability of the Neptune and we contrast our performance with Apache Storm, the dominant stream-processing framework developed by Twitter. At a single node, we are able to achieve a processing rate of ~2 million stream packets per-second. In a distributed setup, we achieved a rate of ~100 million packets per-second.
Author Buddhika, Thilina
Pallickara, Shrideep
Author_xml – sequence: 1
  givenname: Thilina
  surname: Buddhika
  fullname: Buddhika, Thilina
  email: thilinab@cs.colostate.edu
  organization: Dept. of Comput. Sci., Colorado State Univ., Fort Collins, CO, USA
– sequence: 2
  givenname: Shrideep
  surname: Pallickara
  fullname: Pallickara, Shrideep
  email: shrideep@cs.colostate.edu
  organization: Dept. of Comput. Sci., Colorado State Univ., Fort Collins, CO, USA
BookMark eNotjFFLwzAUhSNMcJt79MmX_IHWe5M2bXyT2elgzOK655G2N1pZU0mK4L-3qOfl8B0-zoLN3OCIsRuEGBH03bZ8LA-xAFRxIi_YSmc5pqBBYAJqxuaYSogEZOkVW4TwASBAJnrOqn1RVsd9cc9fyZx51fXED6Mn0_PSDw2F0Lk3bgfPt24k72jkg-XV-7QGblzLD-R-lcJ9dX5wPbkxXLNLa86BVv-9ZMdNUa2fo93L03b9sIs6AfkYkTJWNXXbtLW0aKnWUyZAXbe1zhsJGVhlc025SgSCyqVqDSkltBYtZHLJbv9-OyI6ffquN_77lKWoEFH-AGTmUV4
CODEN IEEPAD
ContentType Conference Proceeding
DBID 6IE
6IL
CBEJK
RIE
RIL
DOI 10.1109/IPDPS.2016.43
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE/IET Electronic Library
IEEE Proceedings Order Plans (POP All) 1998-Present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE
  url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISBN 9781509021406
150902140X
EndPage 1152
ExternalDocumentID 7516111
Genre orig-research
GroupedDBID 29O
6IE
6IF
6IH
6IK
6IL
6IN
AAJGR
ABLEC
ADZIZ
ALMA_UNASSIGNED_HOLDINGS
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CBEJK
CHZPO
IEGSK
IPLJI
JC5
OCL
RIE
RIL
ID FETCH-LOGICAL-i208t-e6af6cbdcdb3f1feb9999cdb19bdb98c3070f6f89e8642106836dae662992d073
IEDL.DBID RIE
ISSN 1530-2075
IngestDate Wed Jun 26 19:23:47 EDT 2024
IsPeerReviewed false
IsScholarly false
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-i208t-e6af6cbdcdb3f1feb9999cdb19bdb98c3070f6f89e8642106836dae662992d073
PageCount 10
ParticipantIDs ieee_primary_7516111
PublicationCentury 2000
PublicationDate 20160501
PublicationDateYYYYMMDD 2016-05-01
PublicationDate_xml – month: 05
  year: 2016
  text: 20160501
  day: 01
PublicationDecade 2010
PublicationTitle 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
PublicationTitleAbbrev IPDPS
PublicationYear 2016
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssj0020349
Score 1.7676395
Snippet Improvements in miniaturization and networking capabilities of sensors have contributed to the proliferation of Internet of Things (IoT) and continuous sensing...
SourceID ieee
SourceType Publisher
StartPage 1143
SubjectTerms Bandwidth
Distributed stream processing
High-throughput data processing
Internet of things
Message systems
Program processors
Real-time systems
Sensors
Throughput
Title NEPTUNE: Real Time Stream Processing for Internet of Things and Sensing Environments
URI https://ieeexplore.ieee.org/document/7516111
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1LS8NAEF5qT56qtuKbPXg0aZp0N4lXjVShJdgWeiv7mICIqUh68dc7k00fiAdv2bAhYYfsfDP7fTOM3cYqMgJDLk-BwAAFIauXDrXyYlvY1EoAYSnfMZ7I0Xz4shCLFrvbamEAoCafgU-X9Vm-XZk1pcr6sUB8QkLegyQInVZrG1xRnRVXGzVAy8diV0-z_5w_5lNicUmfxDl7XVRqJ_LUYePN6x135N1fV9o3378qM_73-45YbyfX4_nWER2zFpQnrLPp18Cb37fLZpMsn80n2T1_RXjISf3B6VRaffBGL4CPc0Sx3OUJoeKrgrvOnlyVlk-J7Y5Tsj11XI_Nn7LZw8hruip4b2GQVB5IVUijrbE6KgYFaISIKQ4GqbY6TQxtAoUskhQSEsEGMomkVSAlOq7Q4o5wytrlqoQzxiGSEAZGI8ZQQwlG2diISOEA1ACUOmddWqTlpyucsWzW5-Lv25fskGzk2IRXrF19reEaPX6lb2pT_wAimauZ
link.rule.ids 310,311,786,790,795,796,802,27956,55107
linkProvider IEEE
linkToHtml http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1NT8JAEJ0QPOgJFYzf7sGjLYXSbetVS0ChaaQk3Mh-TBNjLMaUi7_e2baAMR68dZs22exkd97MznsDcOsLV3kUclkCPQpQCLJa4UAKy9eZDjVH9LTJd0xjPpoPnhbeogF3Wy4MIpbFZ2ibx_IuX6_U2qTKur5H-MQQeffIzzt-xdbahldGaaVSR3XI9r63U9TsjpPHZGbquLht6Dk_-qiUbmTYgulmAlX1yJu9LqStvn5pM_53hofQ2RH2WLJ1RUfQwPwYWpuODazewG1I4yhJ53F0z14IIDLD_2DmXlq8s5oxQL8zwrGsyhRiwVYZq3p7MpFrNjP17vRJ9IMf14H5MEofRlbdV8F67TtBYSEXGVdSKy3drJehJJAY0qAXSi3DQJljIONZEGJgaLAOD1yuBXJOrquv6Uw4gWa-yvEUGLoc-46ShDLEgKMS2leeK2iAoodCnEHbLNLyo5LOWNbrc_736xvYH6XTyXIyjp8v4MDYq6otvIRm8bnGK_L_hbwuzf4NXC2u7Q
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2016+IEEE+International+Parallel+and+Distributed+Processing+Symposium+%28IPDPS%29&rft.atitle=NEPTUNE%3A+Real+Time+Stream+Processing+for+Internet+of+Things+and+Sensing+Environments&rft.au=Buddhika%2C+Thilina&rft.au=Pallickara%2C+Shrideep&rft.date=2016-05-01&rft.pub=IEEE&rft.issn=1530-2075&rft.spage=1143&rft.epage=1152&rft_id=info:doi/10.1109%2FIPDPS.2016.43&rft.externalDocID=7516111
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1530-2075&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1530-2075&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1530-2075&client=summon