DroidRL: Feature selection for android malware detection with reinforcement learning

• Published in: Computers & security Vol. 128; p. 103126
• Main Authors: Wu, Yinwei, Li, Meijin, Zeng, Qi, Yang, Tao, Wang, Junfeng, Fang, Zhiyang, Cheng, Luyu
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.05.2023
• Subjects: Android malware detection; Feature selection; Reinforcement learning; RNN; Sequence processing; Feature selection; RNN; Sequence processing; Android malware detection; Reinforcement learning
• ISSN: 0167-4048; 1872-6208
• DOI: 10.1016/j.cose.2023.103126

•This paper applied reinforcement learning algorithms to the feature selection phase of Android malware detection, reducing the burden of feature selection tasks.•This paper adopts Natural Language Processing methods to tackle the feature selection methods.•This paper presented a modifiable framework that can be easily ported to other feature selection tasks for malware detection. Due to the completely open-source nature of Android, the exploitable vulnerability of malware attacks is increasing. Machine learning, leading to a great evolution in Android malware detection in recent years, is typically applied in the classification phase. Since the correlation between features is ignored in some traditional ranking-based feature selection algorithms, applying wrapper-based feature selection models is a topic worth investigating. Though considering the correlation between features,  wrapper-based approaches are time-consuming for exploring all possible valid feature subsets when processing a large number of Android features. To reduce the computational expense of wrapper-based feature selection, a framework named DroidRL is proposed. The framework deploys DDQN algorithm to obtain a subset of features which can be used for effective malware classification. To select a valid subset of features over a larger range, the exploration-exploitation policy is applied in the model training phase. The recurrent neural network (RNN) is used as the decision network of DDQN to give the framework the ability to sequentially select features. Word embedding is applied for feature representation to enhance the framework’s ability to find the semantic relevance of features. The framework’s feature selection exhibits high performance without any human intervention and can be ported to other feature selection tasks with minor changes. The experiment results show a significant effect when using the Random Forest as DroidRL’s classifier, which reaches 95.6% accuracy with only 24 features selected.