Multilayer ransomware detection using grouped registry key operations, file entropy and file signature monitoring

• Published in: Journal of computer security Vol. 28; no. 3; pp. 337 - 373
• Main Authors: Jethva, Brijesh, Traoré, Issa, Ghaleb, Asem, Ganame, Karim, Ahmed, Sherif
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.01.2020; Sage Publications Ltd
• Subjects: Encryption; Entropy; Machine learning; Malware; Monitoring; Multilayers; Ransomware; Regression analysis; Static code analysis; file signature; machine learning; file entropy; Ransomware detection
• DOI: 10.3233/JCS-191346
• ISSN: 0926-227X

The last few years have come with a sudden rise in ransomware attack incidents, causing significant financial losses to individuals, institutions and businesses. In reaction to these attacks, ransomware detection has become an important topic for research in recent years. Currently, there are two broad categories of ransomware detection techniques: signature-based and behaviour-based analyses. On the one hand, signature-based detection, which mainly relies on a static analysis, can easily be evaded by code-obfuscation and encryption techniques. On the other hand, current behaviour-based models, which rely mainly on a dynamic analysis, face difficulties in accurately differentiating between user-triggered encryption from ransomware-triggered encryption. In the current paper, we present an upgraded behavioural ransomware detection model that reinforces the existing feature space with a new set of features based on grouped registry key operations, introducing a monitoring model based on combined file entropy and file signature. We analyze the new feature model by exploring and comparing three different linear machine learning techniques: SVM, logistic regression and random forest. The proposed approach helps achieve improved detection accuracy and provides the ability to detect novel ransomware. Furthermore, the proposed approach helps differentiate user-triggered encryption from ransomware-triggered encryption, allowing saving as many files as possible during an attack. To conduct our study, we use a new public ransomware detection dataset collected in our lab, which consists of 666 ransomware and 103 benign binaries. Our experimental results show that our proposed approach achieves relatively high accuracy in detecting both previously seen and novel ransomware samples.