Intelligent model for the detection and classification of encrypted network traffic in cloud infrastructure

• Published in: PeerJ. Computer science Vol. 10; p. e2027
• Main Authors: Dawood, Muhammad, Xiao, Chunagbai, Tu, Shanshan, Alotaibi, Faiz Abdullah, Alnfiai, Mrim M, Farhan, Muhammad
• Format: Journal Article
• Language: English
• Published: United States PeerJ Inc 27.05.2024
• Subjects: Artificial Intelligence; Cloud security; Computer Networks and Communications; Cryptography; Intelligent model; Machine learning; SDN; Traffic classification; Traffic classification; Cloud security; SDN; Machine learning; Intelligent model
• ISSN: 2376-5992; 2376-5992
• DOI: 10.7717/peerj-cs.2027

This article explores detecting and categorizing network traffic data using machine-learning (ML) methods, specifically focusing on the Domain Name Server (DNS) protocol. DNS has long been susceptible to various security flaws, frequently exploited over time, making DNS abuse a major concern in cybersecurity. Despite advanced attack, tactics employed by attackers to steal data in real-time, ensuring security and privacy for DNS queries and answers remains challenging. The evolving landscape of internet services has allowed attackers to launch cyber-attacks on computer networks. However, implementing Secure Socket Layer (SSL)-encrypted Hyper Text Transfer Protocol (HTTP) transmission, known as HTTPS, has significantly reduced DNS-based assaults. To further enhance security and mitigate threats like man-in-the-middle attacks, the security community has developed the concept of DNS over HTTPS (DoH). DoH aims to combat the eavesdropping and tampering of DNS data during communication. This study employs a ML-based classification approach on a dataset for traffic analysis. The AdaBoost model effectively classified Malicious and Non-DoH traffic, with accuracies of 75% and 73% for DoH traffic. The support vector classification model with a Radial Basis Function (SVC-RBF) achieved a 76% accuracy in classifying between malicious and non-DoH traffic. The quadratic discriminant analysis (QDA) model achieved 99% accuracy in classifying malicious traffic and 98% in classifying non-DoH traffic.