An information-theoretic perspective of tf–idf measures

• Published in: Information processing & management Vol. 39; no. 1; pp. 45 - 65
• Main Author: Aizawa, Akiko
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 2003; Elsevier Science; Elsevier Science Ltd
• Subjects: Document Handling; Exact sciences and technology; Frequency Data; Information and communication sciences; Information management; Information Retrieval; Information retrieval systems; Information retrieval systems. Information and document management system; Information science. Documentation; Information theory; Information theory approach; Mathematical Formulas; Measurement Techniques; Online information retrieval; Probability; Probability weighted amount of information; Sciences and techniques of general use; Searching; Specificity; Studies; Subject Index Terms; Term weighting theories; Text categorization; tf–idf; Theoretical Analysis; Theory; Weighted Data; Text categorization; Term weighting theories; tf–idf; Information theory; Word; Weighting; Measurement indicator; Occurrence frequency; Search system; Probability; Information retrieval; Algorithm; Categorization; Statistics

This paper presents a mathematical definition of the “probability-weighted amount of information” (PWI), a measure of specificity of terms in documents that is based on an information-theoretic view of retrieval events. The proposed PWI is expressed as a product of the occurrence probabilities of te...