Continuous monitoring for changepoints in data streams using adaptive estimation

• Published in: Statistics and computing Vol. 27; no. 5; pp. 1257 - 1270
• Main Authors: Bodenham, Dean A., Adams, Niall M.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.09.2017; Springer Nature B.V
• Subjects: Adaptive control; Artificial Intelligence; Change detection; Computer simulation; Data transmission; Mathematics and Statistics; Monitoring; Probability and Statistics in Computer Science; Statistical Theory and Methods; Statistics; Statistics and Computing/Statistics Programs; Sequential analysis; Changepoint detection; Data stream; Adaptive estimation
• ISSN: 0960-3174; 1573-1375
• DOI: 10.1007/s11222-016-9684-8

Data streams are characterised by a potentially unending sequence of high-frequency observations which are subject to unknown temporal variation. Many modern streaming applications demand the capability to sequentially detect changes as soon as possible after they occur, while continuing to monitor the stream as it evolves. We refer to this problem as continuous monitoring . Sequential algorithms such as CUSUM, EWMA and their more sophisticated variants usually require a pair of parameters to be selected for practical application. However, the choice of parameter values is often based on the anticipated size of the changes and a given choice is unlikely to be optimal for the multiple change sizes which are likely to occur in a streaming data context. To address this critical issue, we introduce a changepoint detection framework based on adaptive forgetting factors that, instead of multiple control parameters, only requires a single parameter to be selected. Simulated results demonstrate that this framework has utility in a continuous monitoring setting. In particular, it reduces the burden of selecting parameters in advance. Moreover, the methodology is demonstrated on real data arising from Foreign Exchange markets.