Dache: A data aware caching for big-data applications using the MapReduce framework
The buzz-word big-data (application) refers to the large-scale distributed applications that work on unprecedentedly large data sets. Google's MapReduce framework and Apache's Hadoop, its open-source implementation, are the defacto software system for big-data applications. An observation...
Saved in:
Published in | 2013 Proceedings IEEE INFOCOM pp. 35 - 39 |
---|---|
Main Authors | , |
Format | Conference Proceeding |
Language | English |
Published |
IEEE
01.04.2013
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | The buzz-word big-data (application) refers to the large-scale distributed applications that work on unprecedentedly large data sets. Google's MapReduce framework and Apache's Hadoop, its open-source implementation, are the defacto software system for big-data applications. An observation regarding these applications is that they generate a large amount of intermediate data, and these abundant information is thrown away after the processing finish. Motivated by this observation, we propose a data-aware cache framework for big-data applications, which is called Dache. In Dache, tasks submit their intermediate results to the cache manager. A task, before initiating its execution, queries the cache manager for potential matched processing results, which could accelerate its execution or even completely saves the execution. A novel cache description scheme and a cache request and reply protocol are designed. We implement Dache by extending the relevant components of the Hadoop project. Testbed experiment results demonstrate that Dache significantly improves the completion time of MapReduce jobs and saves a significant chunk of CPU execution time. |
---|---|
ISBN: | 9781467359443 1467359440 |
ISSN: | 0743-166X 2641-9874 |
DOI: | 10.1109/INFCOM.2013.6566730 |