CoCITe-Coordinating Changes in Text

• Published in: IEEE transactions on knowledge and data engineering Vol. 24; no. 1; pp. 15 - 29
• Main Authors: Wright, J. H., Grothendieck, J.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.01.2012; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Data models; Dynamic programming; Heuristic algorithms; modeling structured; Multimedia communication; Statistical analysis; Statistical software; Text mining; textual and multimedia data; Time frequency analysis
• DOI: 10.1109/TKDE.2010.250
• ISSN: 1041-4347

Text streams are ubiquitous and contain a wealth of information, but are typically orders of magnitude too large in scale for comprehensive human inspection. There is a need for tools that can detect and group changes occurring within text streams and substreams, in order to find, structure, and summarize these changes for presentation to human analysts. This paper describes a procedure for efficiently finding step changes, trends, bursts, and cyclic changes affecting frequencies of words, or more general lexical items, within streams of documents which may be optionally labeled with metadata. The common phenomenon of over-dispersion is accommodated using mixture distributions. A streaming implementation is described which can process data from a continuous feed. Anomalies can be detected, grouped, and rendered visually for human comprehension.