A Novel Method for Detecting Future Generations of Targeted and Metamorphic Malware Based on Genetic Algorithm

• Published in: IEEE access Vol. 9; pp. 69951 - 69970
• Main Authors: Javaheri, Danial, Lalbakhsh, Pooia, Hosseinzadeh, Mehdi
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial neural networks; Crossovers; cyber security; data mining; Emulators; Engines; Feature extraction; Genes; genetic algorithm; Genetic algorithms; Machine learning; Malware; Malware detection; malware unpacking; metamorphism; Modelling; Monitoring; obfuscation; Optimization; Performance evaluation; Permutations
• DOI: 10.1109/ACCESS.2021.3077295
• ISSN: 2169-3536

This paper presents a novel solution for detecting rare and mutating malware programs and provides a strategy to address the scarcity of datasets for modeling these types of malware. To provide sufficient training data for malware behavioral modeling, genetic algorithms are used together with an optimization strategy that selectively creates generations of mutated elite malware samples. In our unique method, a sequence of system API calls is extracted using tracker filter drivers in a sandbox environment. The most obfuscated and metamorphic malware are chosen by an elite selection method. The behavioral chromosomes are formed by mapping extracted APIs to genes using linear regression. Our analysis system includes an Internet simulator and a human emulator to deceive intelligent classes of malware to successfully execute themselves and prevent system halting. The evolution process is performed through crossover and permutation of genes, which are encoded based on the addresses of the kernel-level system functions. An objective function has been defined to optimize the vital indicators of malignancy and tracking rate with a linear time complexity. This guarantees that new generations of malware are more destructive and stealthy than their parents. J48 and deep neural networks were employed in our experiments as they are two popular modeling and classification strategies in the area of behavioral malware detection. Real-world malware samples from valid references were used for the performance evaluation of our approach. Comprehensive scenarios were involved in the experiments to evaluate the performance of our proposed strategy. The results demonstrate significant improvement in detection accuracy - up to 5% considering rare and metamorphic malware. The results also demonstrated a considerable enhancement in true positive rate for the proposed deep-learning algorithm.