AB-DHD: An Attention Mechanism and Bi-Directional Gated Recurrent Unit Based Model for Dynamic Link Library Hijacking Vulnerability Discovery

• Published in: Journal of computer science and technology Vol. 40; no. 3; pp. 887 - 903
• Main Authors: Chen, Xiao, Sha, Le-Tian, Xiao, Fu, Pan, Jia-Ye, Dong, Jian-Kuo
• Format: Journal Article
• Language: English
• Published: Singapore Springer Nature Singapore 01.05.2025; Springer Nature B.V
• Subjects: Artificial Intelligence; Computer Science; Data processing; Data Structures and Information Theory; Datasets; Dynamic link libraries; Information Systems Applications (incl.Internet); Neural networks; Regular Paper; Software Engineering; Theory of Computation; Vulnerability; dynamic link library hijacking; attention mechanism; operating system; vulnerability discovery; gated recurrent unit (GRU)
• ISSN: 1000-9000; 1860-4749
• DOI: 10.1007/s11390-025-4497-x

With the rapid development of operating systems, attacks on system vulnerabilities are increasing. Dynamic link library (DLL) hijacking is prevalent in installers on freeware platforms and is highly susceptible to exploitation by malware attackers. However, existing studies are based solely on the load paths of DLLs, ignoring the attributes of installers and invocation modes, resulting in low accuracy and weak generality of vulnerability detection. In this paper, we propose a novel model, AB-DHD, which is based on an attention mechanism and a bi-directional gated recurrent unit (BiGRU) neural network for DLL hijacking vulnerability discovery. While BiGRU is an enhancement of GRU and has been widely applied in sequence data processing, a double-layer BiGRU network is introduced to analyze the internal features of installers with DLL hijacking vulnerabilities. Additionally, an attention mechanism is incorporated to dynamically adjust feature weights, significantly enhancing the ability of our model to detect vulnerabilities in new installers. A comprehensive “List of Easily Hijacked DLLs” is developed to serve a reference for future studies. We construct an EXEFul dataset and a DLLVul dataset, using data from two publicly available authoritative vulnerability databases, Common Vulnerabilities & Exposures (CVE) and China National Vulnerability Database (CNVD), and mainstream installer distribution platforms. Experimental results show that our model outperforms popular automated tools like Rattler and DLLHSC, achieving an accuracy of 97.79% and a recall of 94.72%. Moreover, 17 previously unknown vulnerabilities have been identified, and corresponding vulnerability certifications have been assigned.