Road Traffic Event Detection Using Twitter Data, Machine Learning, and Apache Spark

• Published in: 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) pp. 1888 - 1895
• Main Authors: Alomari, Ebtesam, Mehmood, Rashid, Katib, Iyad
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.08.2019
• Subjects: Apache Spark; Big Data; Big data analytics; Event detection; Logistic Regressio; Machine learning; MongoDB; Naïve Bayes; Roads; Smart cities; Smart transportation; Support Vector Machine (SVM); Support vector machines; Text mining; Twitter; Twitter data analysis
• DOI: 10.1109/SmartWorld-UIC-ATC-SCALCOM-IOP-SCI.2019.00332

Road transportation is the backbone of modern societies, yet it costs annually over a million deaths and trillions of dollars to the global economy. Social media such as Twitter have increasingly become an important source of information in many dimensions of smart societies. Automatic detection of road traffic events using Twitter data mining is one such area of a great many applications and enormous potential, albeit facing major challenges concerning the management and analysis of big data (volume, velocity, variety, and veracity). Various approaches on the subject have been proposed in recent years, but the methods and outcomes are in their infancy. This paper proposes a method for automatic detection of road traffic related events from tweets in the Saudi dialect using machine learning and big data technologies. Firstly, we build and train a classifier using three machine learning algorithms, Naïve Bayes, Support Vector Machine, and logistic regression, to filter tweets into relevant and irrelevant. Subsequently, we train other classifiers to detect multiple types of events including accident, roadwork, road closure, road damage, traffic condition, fire, weather, and social events. The results from the analysis of one million tweets show that our method is able to detect road traffic events, as well as their location and time, automatically, without any prior knowledge of the events. To the best of our knowledge, this is the first work on traffic event detection from Arabic tweets using machine learning and the Apache Spark big data platform.