Kahuna: Problem diagnosis for Mapreduce-based cloud computing environments

We present Kahuna, an approach that aims to diagnose performance problems in MapReduce systems. Central to Kahuna's approach is our insight on peer-similarity, that nodes behave alike in the absence of performance problems, and that a node that behaves differently is the likely culprit of a per...

Full description

Saved in:
Bibliographic Details
Published in2010 IEEE Network Operations and Management Symposium - NOMS 2010 pp. 112 - 119
Main Authors Jiaqi Tan, Xinghao Pan, Marinelli, Eugene, Kavulya, Soila, Gandhi, Rajeev, Narasimhan, Priya
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.04.2010
Subjects
Cloud computing
Clustering algorithms
Data mining
Facebook
Fault diagnosis
Large-scale systems
Measurement
Open source software
Peer to peer computing
Statistics
Online AccessGet full text

Cover

More Information
Summary:We present Kahuna, an approach that aims to diagnose performance problems in MapReduce systems. Central to Kahuna's approach is our insight on peer-similarity, that nodes behave alike in the absence of performance problems, and that a node that behaves differently is the likely culprit of a performance problem. We present applications of Kahuna's insight in techniques and their algorithms to statistically compare black-box (OS-level performance metrics) and white-box (Hadoop-log statistics) data across the different nodes of a MapReduce cluster, in order to identify the faulty node(s). We also present empirical evidence of our peer-similarity observations from the 4000-processor Yahoo! M45 Hadoop cluster. In addition, we demonstrate Kahuna's effectiveness through experimental evaluation of two algorithms for a number of reported performance problems, on four different workloads in a 100-node Hadoop cluster running on Amazon's EC2 infrastructure.
ISBN:9781424453665
1424453666
ISSN:1542-1201
2374-9709
DOI:10.1109/NOMS.2010.5488446