Machine Learning Predictions of Runtime and IO Traffic on High-End Clusters

We use supervised machine learning algorithms (i.e., Decision Trees, Random Forest, and K-nearest Neighbors) to predict performance characteristics such as runtime and IO traffic of batch jobs on high-end clusters, using only user job scripts as input. We show that decision trees outperform other al...

Full description

Saved in:
Bibliographic Details
Published in2016 IEEE International Conference on Cluster Computing (CLUSTER) pp. 255 - 258
Main Authors McKenna, Ryan, Herbein, Stephen, Moody, Adam, Gamblin, Todd, Taufer, Michela
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.09.2016
Subjects
Decision trees
Machine learning algorithms
Measurement
Monitoring
Runtime
Training
Training data
Online AccessGet full text
ISSN2168-9253
DOI10.1109/CLUSTER.2016.58

Cover

More Information
Summary:We use supervised machine learning algorithms (i.e., Decision Trees, Random Forest, and K-nearest Neighbors) to predict performance characteristics such as runtime and IO traffic of batch jobs on high-end clusters, using only user job scripts as input. We show that decision trees outperform other algorithms and accurately predict the runtime of 73% of jobs within a error tolerance of 10 minutes, which is a 51% improvement over the user requested runtime.
ISSN:2168-9253
DOI:10.1109/CLUSTER.2016.58