Enhanced PeerHunter: Detecting Peer-to-Peer Botnets Through Network-Flow Level Community Behavior Analysis

• Published in: IEEE transactions on information forensics and security Vol. 14; no. 6; pp. 1485 - 1500
• Main Authors: Di Zhuang, Chang, J. Morris
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Botnet; Communities; community detection; Cryptography; Cybercrime; Cybersecurity; Feature extraction; Indexes; intrusion detection; Machine learning algorithms; Malware; network security; P2P botnet; Payloads; Peer to peer computing; Social networks
• ISSN: 1556-6013; 1556-6021
• DOI: 10.1109/TIFS.2018.2881657

Peer-to-peer (P2P) botnets have become one of the major threats in network security for serving as the fundamental infrastructure for various cyber-crimes. More challenges are involved in the problem of detecting P2P botnets, despite a few work claimed to detect centralized botnets effectively. We propose an enhanced PeerHunter, a network-flow level community behavior analysis based system, to detect P2P botnets. Our system starts from a P2P network flow detection component. Then, it uses "mutual contacts" to cluster bots into communities. Finally, it uses network-flow level community behavior analysis to detect potential botnets. In the experimental evaluation, we propose two evasion attacks, where we assume the adversaries know our techniques in advance and attempt to evade our system by making the P2P bots mimic the behavior of legitimate P2P applications. Our results showed that enhanced PeerHunter can obtain high detection rate with few false positives, and high robustness against the proposed attacks.