A novel framework for generic Spark workload characterization and similar pattern recognition using machine learning

• Published in: Journal of parallel and distributed computing Vol. 189; p. 104881
• Main Authors: Garralda-Barrio, Mariano, Eiras-Franco, Carlos, Bolón-Canedo, Verónica
• Format: Journal Article
• Language: English
• Published: Elsevier Inc 01.07.2024
• Subjects: Apache spark; Big data; Machine learning; Pattern recognition; Workload characterization; Big data; Pattern recognition; Apache spark; Workload characterization; Machine learning
• ISSN: 0743-7315; 1096-0848
• DOI: 10.1016/j.jpdc.2024.104881

Comprehensive workload characterization plays a pivotal role in comprehending Spark applications, as it enables the analysis of diverse aspects and behaviors. This understanding is indispensable for devising downstream tuning objectives, such as performance improvement. To address this pivotal issue, our work introduces a novel and scalable framework for generic Spark workload characterization, complemented by consistent geometric measurements. The presented approach aims to build robust workload descriptors by profiling only quantitative metrics at the application task-level, in a non-intrusive manner. We expand our framework for downstream workload pattern recognition by incorporating unsupervised machine learning techniques: clustering algorithms and feature selection. These techniques significantly improve the process of grouping similar workloads without relying on predefined labels. We effectively recognize 24 representative Spark workloads from diverse domains, including SQL, machine learning, web search, graph, and micro-benchmarks, available in HiBench. Our framework achieves a high accuracy F-Measure score of up to 90.9% and a Normalized Mutual Information of up to 94.5% in similar workload pattern recognition. These scores significantly outperform the results obtained in a comparative analysis with an established workload characterization approach in the literature. •Building robust and scalable workload vector descriptors in a generic manner.•Recognizing workload patterns in latent space using unsupervised ML techniques.•Evaluates 24 Spark workloads with geometric, internal, and external metrics.•Clustering attains F-Measure up to 90.9% and NMI up to 94.5%.