Novelty detection for text documents using named entity recognition

• Published in: 2007 6th International Conference on Information, Communications and Signal Processing pp. 1 - 5
• Main Authors: Kok Wah Ng, Tsai, F.S., Lihui Chen, Kiat Chong Goh
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2007
• Subjects: Data mining; Equations; Frequency; Information processing; novelty detection; recommender system; Recommender systems; Speech; Tagging; Testing; text mining; Text recognition
• ISBN: 1424409829; 9781424409822
• DOI: 10.1109/ICICS.2007.4449883

In order to determine novel information from raw text documents, a novelty detection recommender system was developed to explore the method of comparing various types of entities within sentences. We first detected novel sentences using named entity recognition to extract the entity types of person, place, time, and organization. In addition, part-of-speech tagging was performed to tag each word in the documents, allowing syntactic structures of noun, verb, and adjective to be used for comparisons. WordNet, an English lexical database of concepts and relations, was also incorporated to generate synonyms for the entities and parts of speech as well as to determine the similarity of sentences. The novelty score of each sentence was determined by using two different metrics, UniqueComparison and Importance Value. UniqueComparison calculated the number of matched entities, whereas ImportanceValue took into account the total weight of matched words that coexisted in both the test and history sentences. The results look promising when compared to the benchmark scores for the Text Retrieval Conference's (TREC) Novelty Track 2004. This demonstrated that the combination of named entity recognition and part-of-speech tagging is capable of detecting novelty with good results.