Eager Memory Management for In-Memory Data Analytics

This paper introduces e-spill, an eager spill mechanism, which dynamically finds the optimal spill-threshold by monitoring the GC time at runtime and thereby prevent expensive GC overhead. Our e-spill adopts a slow-start model to gradually increase the spill-threshold until it reaches the optimal po...

Full description

Saved in:
Bibliographic Details
Published inIEICE Transactions on Information and Systems Vol. E102.D; no. 3; pp. 632 - 636
Main Authors JANG, Hakbeom, BAE, Jonghyun, HAM, Tae Jun, LEE, Jae W.
Format Journal Article
LanguageEnglish
Published Tokyo The Institute of Electronics, Information and Communication Engineers 01.03.2019
Japan Science and Technology Agency
Subjects
Analytics
data spill
garbage collection
in-memory computing
Memory management
Performance enhancement
spark
Online AccessGet full text

Cover

More Information
Summary:This paper introduces e-spill, an eager spill mechanism, which dynamically finds the optimal spill-threshold by monitoring the GC time at runtime and thereby prevent expensive GC overhead. Our e-spill adopts a slow-start model to gradually increase the spill-threshold until it reaches the optimal point without substantial GCs. We prototype e-spill as an extension to Spark and evaluate it using six workloads on three different parallel platforms. Our evaluations show that e-spill improves performance by up to 3.80× and saves the cost of cluster operation on Amazon EC2 cloud by up to 51% over the baseline system following Spark Tuning Guidelines.
ISSN:0916-8532
1745-1361
DOI:10.1587/transinf.2018EDL8199