A framework for understanding Latent Semantic Indexing (LSI) performance

• Published in: Information processing & management Vol. 42; no. 1; pp. 56 - 73
• Main Authors: Kontostathis, April, Pottenger, William M.
• Format: Journal Article
• Language: English
• Published: Oxford Elsevier Ltd 2006; Elsevier Science; Elsevier Science Ltd
• Subjects: Algorithms; Correlation analysis; Decomposition; Exact sciences and technology; Indexing; Information and communication sciences; Information processing; Information processing and retrieval; Information retrieval; Information retrieval theory; Information retrieval. Man machine relationship; Information science. Documentation; Latent Semantic Indexing; Research process. Evaluation; Sciences and techniques of general use; Semantics; Singular value; Studies; Subject indexing; Term co-occurrence; Terms; Latent Semantic Indexing; Decomposition; Information retrieval theory; Term co-occurrence; Singular value; Theoretical model; Semantics; Information retrieval; Performance; Search algorithm; Singular value decomposition; Indexing
• ISSN: 0306-4573; 1873-5371
• DOI: 10.1016/j.ipm.2004.11.007

In this paper we present a theoretical model for understanding the performance of Latent Semantic Indexing (LSI) search and retrieval application. Many models for understanding LSI have been proposed. Ours is the first to study the values produced by LSI in the term by dimension vectors. The framework presented here is based on term co-occurrence data. We show a strong correlation between second-order term co-occurrence and the values produced by the Singular Value Decomposition (SVD) algorithm that forms the foundation for LSI. We also present a mathematical proof that the SVD algorithm encapsulates term co-occurrence information.