Cross-Correlation as Tool to Determine the Similarity of Series of Measurements for Big-Data Analysis Tasks

• Published in: Cloud Computing and Big Data pp. 263 - 282
• Main Authors: Hilbrich, Marcus, Müller-Pfefferkorn, Ralph
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Analysis; Big-data; Cloud; Cross-correlation; Grid; HPC; Jobs; Monitoring; Normalisation; Series of measurements; Similarity; Smart-data
• ISBN: 9783319284293; 3319284290
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-28430-9_20

One aspect of the so called Big Data challenge is the rising quantity of data in almost all scientific, social, governmental and commercial disciplines. As a result there are many ongoing developments of analysis techniques to substitute manual processes with automatic or semi-automatic algorithms. This means the knowledge of data analysts has to be transferred to algorithms which can be executed simultaneously on many data sets. Such, the rising amount of data can be analysed in an constant quality and in a shorter time. Even if the number of existing algorithms is enormous, a ready to use solution for each problem doesn’t exist. Especially for analysing and comparing series of measurements, e.g. for analysing data of activity trackers or to monitor service execution infrastructures, we discovered a lack of options. Thus we explain the basics of an algorithm using the cross-correlation function to determine a meaningful value of similarity for two or more series of measurements. We used the new method to analyse and categorise job centric monitoring data.