Real-time event detection from the Twitter data stream using the TwitterNews+ Framework

• Published in: Information processing & management Vol. 56; no. 3; pp. 1146 - 1165
• Main Authors: Hasan, Mahmud, Orgun, Mehmet A., Schwitter, Rolf
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier Ltd 01.05.2019; Elsevier Sequoia S.A
• Subjects: Clustering; Event detection; Incremental clustering; Microblog; Parameter sensitivity analysis; Real time; Sensitivity analysis; Social media; Social networks; Twitter; Websites; Incremental clustering; Twitter; Microblog; Event detection; Parameter sensitivity analysis; Social media
• ISSN: 0306-4573; 0166-0462; 1873-5371; 1879-2308
• DOI: 10.1016/j.ipm.2018.03.001

Detecting events in real-time from the Twitter data stream has gained substantial attention in recent years from researchers around the world. Different event detection approaches have been proposed as a result of these research efforts. One of the major challenges faced in this context is the high computational cost associated with event detection in real-time. We propose, TwitterNews+, an event detection system that incorporates specialized inverted indices and an incremental clustering approach to provide a low computational cost solution to detect both major and minor newsworthy events in real-time from the Twitter data stream. In addition, we conduct an extensive parameter sensitivity analysis to fine-tune the parameters used in TwitterNews+ to achieve the best performance. Finally, we evaluate the effectiveness of our system using a publicly available corpus as a benchmark dataset. The results of the evaluation show a significant improvement in terms of recall and precision over five state-of-the-art baselines we have used.