An Improved System For Sentence-Level Novelty Detection In Textual Streams

• Published in: 2015 International Conference on Smart and Sustainable City and Big Data (ICSSC) p. 6
• Main Authors: Xinyu Fu, Ch'ng, E, Aickelin, U, Lanyun Zhang
• Format: Conference Proceeding
• Language: English
• Published: Stevenage, UK IET 2015; The Institution of Engineering & Technology
• Subjects: Adaptation; Data handling techniques; Data transmission; File organisation; Information analysis and indexing; Information networks; Knowledge engineering techniques; News; Other topics in statistics; Streams; Vector space; Weighting; LSH; text analysis; Google news; locality sensitive hashing; data mining; probability; vector space model; data stream; miss probability; sentence-level novelty detection; incremental term frequency-inverse document frequency weighting; WWW; file organisation; TF-IDF weighting; novelty detection framework; event detection system; Internet; textual stream; baseline system
• ISBN: 9781785610325; 1785610325
• DOI: 10.1049/cp.2015.0250

Novelty detection in news events has long been a difficult problem. A number of models performed well on specific data streams but certain issues are far from being solved, particularly in large data streams from the WWW where unpredictability of new terms requires adaptation in the vector space model. We present a novel event detection system based on the Incremental Term Frequency-Inverse Document Frequency (TF-IDF) weighting incorporated with Locality Sensitive Hashing (LSH). Our system could efficiently and effectively adapt to the changes within the data streams of any new terms with continual updates to the vector space model. Regarding miss probability, our proposed novelty detection framework outperforms a recognised baseline system by approximately 16% when evaluating a benchmark dataset from Google News.