A Needle is an Outlier in a Haystack: Hunting Malicious PyPI Packages with Code Clustering

• Published in: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) pp. 307 - 318
• Main Authors: Liang, Wentao, Ling, Xiang, Wu, Jingzheng, Luo, Tianyue, Wu, Yanjun
• Format: Conference Proceeding
• Language: English
• Published: IEEE 11.09.2023
• Subjects: code clustering; Ecosystems; malicious package detection; Malware; Manuals; Needles; Pressing; PyPI; Rendering (computer graphics); Supply chains
• ISSN: 2643-1572
• DOI: 10.1109/ASE56229.2023.00085

As the most popular Python software repository, PyPI has become an indispensable part of the Python ecosystem. Regrettably, the open nature of PyPI exposes end-users to substantial security risks stemming from malicious packages. Consequently, the timely and effective identification of malware within the vast number of newly-uploaded PyPI packages has emerged as a pressing concern. Existing detection methods are dependent on difficult-to-obtain explicit knowledge, such as taint sources, sinks, and malicious code patterns, rendering them susceptible to overlooking emergent malicious packages. In this paper, we present a lightweight and effective method, namely MPHunter, to detect malicious packages without requiring any explicit prior knowledge. MPHunter is founded upon two fundamental and insightful observations. First, malicious packages are considerably rarer than benign ones, and second, the functionality of installation scripts for malicious packages diverges significantly from those of benign packages, with the latter frequently forming clusters. Consequently, MPHunter utilizes clustering techniques to group the installation scripts of PyPI packages and identifies outliers. Subsequently, MPHunter ranks the outliers according to their outlierness and the distance between them and known malicious instances, thereby effectively highlighting potential evil packages. With MPHunter, we successfully identified 60 previously unknown malicious packages from a pool of 31,329 newly-uploaded packages over a two-month period. All of them have been confirmed by the PyPI official. Moreover, a manual analysis shows that MPHunter recognizes all potentially malicious installation scripts with a recall of 100% across all analyzed packages. We assert that MPHunter offers a valuable and advantageous supplement to existing detection techniques, augmenting the arsenal of software supply chain security analysis.