A study of dynamic meta-learning for failure prediction in large-scale systems

Despite years of study on failure prediction, it remains an open problem, especially in large-scale systems composed of vast amount of components. In this paper, we present a dynamic meta-learning framework for failure prediction. It intends to not only provide reasonable prediction accuracy, but al...

Full description

Saved in:
Bibliographic Details
Published inJournal of parallel and distributed computing Vol. 70; no. 6; pp. 630 - 643
Main Authors Lan, Zhiling, Gu, Jiexing, Zheng, Ziming, Thakur, Rajeev, Coghlan, Susan
Format Journal Article
LanguageEnglish
Published Amsterdam Elsevier Inc 01.06.2010
Elsevier
Subjects
Accuracy
Applied sciences
Blue Gene
Computer science; control theory; systems
Computer systems performance. Reliability
Dynamic techniques
Dynamic tests
Dynamical systems
Dynamics
Exact sciences and technology
Failure
Failure prediction
Genes
Large-scale systems
Logs
Meta-learning
Monitoring
Software
Supercomputers
Dynamic techniques
Blue Gene
Failure prediction
Large-scale systems
Meta-learning
Supercomputer
Log file
Forecasting
Monitoring
Large scale system
Online AccessGet full text

Cover

More Information
Summary:Despite years of study on failure prediction, it remains an open problem, especially in large-scale systems composed of vast amount of components. In this paper, we present a dynamic meta-learning framework for failure prediction. It intends to not only provide reasonable prediction accuracy, but also be of practical use in realistic environments. Two key techniques are developed to address technical challenges of failure prediction. One is meta-learning to boost prediction accuracy by combining the benefits of multiple predictive techniques. The other is a dynamic approach to dynamically obtain failure patterns from a changing training set and to dynamically extract effective rules by actively monitoring prediction accuracy at runtime. We demonstrate the effectiveness and practical use of this framework by means of real system logs collected from the production Blue Gene/L systems at Argonne National Laboratory and San Diego Supercomputer Center. Our case studies indicate that the proposed mechanism can provide reasonable prediction accuracy by forecasting up to 82% of the failures, with a runtime overhead less than 1.0 min.
Bibliography:ObjectType-Article-2
SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 23
ISSN:0743-7315
1096-0848
DOI:10.1016/j.jpdc.2010.03.003