A new method for tuning the CNN pre-trained models as a feature extractor for malware detection

• Published in: Pattern analysis and applications : PAA Vol. 28; no. 1
• Main Author: Bakır, Halit
• Format: Journal Article
• Language: English
• Published: London Springer London 01.03.2025; Springer Nature B.V
• Subjects: Accuracy; Artificial neural networks; Bayesian analysis; Computer Science; Configuration management; Cybersecurity; Machine learning; Malware; Optimization; Original Paper; Pattern Recognition; Robustness; Tactics; Tuning; Pre-trained model; Hyperparameters tuning; Malware detection; Convolutional neural network; Android
• ISSN: 1433-7541; 1433-755X
• DOI: 10.1007/s10044-024-01381-x

Despite significant advancements in Android malware detection, current approaches face notable challenges, particularly in handling obfuscation techniques, achieving high detection accuracy, and maintaining computational efficiency. Traditional static and dynamic analysis methods often struggle with evolving malware tactics and providing lightweight execution, which necessitates models that can dynamically adapt to these challenges. To address these needs, this study presents TuneDroid, a novel approach that optimizes CNN model configurations for both improved detection rates and resilience to obfuscation. By leveraging image-based visualization of code, TuneDroid enables CNNs to recognize high-level visual patterns that remain consistent even with code modifications, thereby enhancing robustness against common evasion tactics. TuneDroid utilizes Bayesian optimization for dynamically tuning pre-trained Convolutional Neural Network (CNN) models. This optimization process selects optimal pre-trained models, layer configurations, and positions, significantly enhancing detection performance. Using a dataset of 3000 benign and 3000 malicious apps, where DEX code is converted into images, TuneDroid achieved accuracy rates of 99.44% on the validation set and 98.00% on the testing set. In comparison, static end-to-end models without hyperparameter tuning yielded lower accuracies, not exceeding 90.50% and 91.17%. The robustness of TuneDroid’s performance is demonstrated through extensive experiments, including precision, recall, F1-score, and comparisons with baseline models. These results highlight the importance of dynamic tuning in maximizing the effectiveness of CNN-based malware detection. This work stands out by focusing on the dynamic tuning of deep learning models for Android app security, demonstrating substantial improvements in detection accuracy and showcasing the potential of Bayesian optimization in this context.