Analyzing Big Data Originated from Data Communication Networks using K-Means Algorithm to Understand the Nature of Incoming Malicious Connections

An environment is developed with a distributed Apache SPARK, deployed on Hadoop cluster for timely inference and classification of security incidents. Analysis of 85GB of network-packet dataset collected over four months is done (provided by CSIR-4PI, Govt. of India). K-means machine learning algori...

Full description

Saved in:
Bibliographic Details
Published in2020 Third International Conference on Multimedia Processing, Communication & Information Technology (MPCIT) pp. 129 - 132
Main Authors Shyamasundar, L B, Kumar, V Anil, Prathuri, Jhansi Rani
Format Conference Proceeding
LanguageEnglish
Published IEEE 11.12.2020
Subjects
Big Data
Data communication
Data visualization
Entropy
Information technology
K-means clustering
Machine Learning
Machine learning algorithms
Network Trace Data Analysis
Principal Component Analysis(PCA)
Security
Telecommunication traffic
Online AccessGet full text

Cover

More Information
Summary:An environment is developed with a distributed Apache SPARK, deployed on Hadoop cluster for timely inference and classification of security incidents. Analysis of 85GB of network-packet dataset collected over four months is done (provided by CSIR-4PI, Govt. of India). K-means machine learning algorithm is used to analyze the network traffic based on various fields. By building and evaluating models, optimum number of clusters was determined. Clustering results are evaluated by calculating the clustering score using Within-Set Sum-of-Squared-Errors(WSSSE), entropy, Silhotte, Davies-Bouldin-Index and Dunn-Index. Several plots are visualized to understand the clustering analysis results and understand the nature of incoming malicious connections.
DOI:10.1109/MPCIT51588.2020.9350510