Mining world knowledge for analysis of search engine content

• Published in: Web intelligence and agent systems Vol. 5; no. 3; pp. 233 - 253
• Main Authors: King, John D., Li, Yuefeng, Tao, Xiaohui, Nayak, Richi
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.08.2007
• Subjects: search engines; hierarchal classification; taxonomy; collection selection; Ontology; data mining
• ISSN: 1570-1263; 1875-9289
• DOI: 10.3233/WEB-2007-wia00115

Little is known about the content of the major search engines. We present an automatic learning method which trains an ontology with world knowledge of hundreds of different subjects in a three-level taxonomy covering the documents offered in our university library. We then mine this ontology to find important classification rules, and then use these rules to perform an extensive analysis of the content of the largest general purpose internet search engines in use today. Instead of representing documents and collections as a set of terms, we represent them as a set of subjects, which is a highly efficient representation, leading to a more robust representation of information and a decrease of synonymy.