Efficient and flexible algorithms for monitoring distance-based outliers over data streams

• Published in: Information systems (Oxford) Vol. 55; pp. 37 - 53
• Main Authors: Kontaki, Maria, Gounaris, Anastasios, Papadopoulos, Apostolos N., Tsichlas, Kostas, Manolopoulos, Yannis
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.01.2016
• Subjects: Algorithms; Data analysis; Data transmission; Flexibility; Information systems; Monitoring; Outlier detection; Outliers (statistics); Stream data mining; Tasks; Stream data mining; Outlier detection
• ISSN: 0306-4379; 1873-6076
• DOI: 10.1016/j.is.2015.07.006

Anomaly detection is considered an important data mining task, aiming at the discovery of elements (known as outliers) that show significant diversion from the expected case. More specifically, given a set of objects the problem is to return the suspicious objects that deviate significantly from the typical behavior. As in the case of clustering, the application of different criteria leads to different definitions for an outlier. In this work, we focus on distance-based outliers: an object x is an outlier if there are less than k objects lying at distance at most R from x. The problem offers significant challenges when a stream-based environment is considered, where data arrive continuously and outliers must be detected on-the-fly. There are a few research works studying the problem of continuous outlier detection. However, none of these proposals meets the requirements of modern stream-based applications for the following reasons: (i) they demand a significant storage overhead, (ii) their efficiency is limited and (iii) they lack flexibility in the sense that they assume a single configuration of the k and R parameters. In this work, we propose new algorithms for continuous outlier monitoring in data streams, based on sliding windows. Our techniques are able to reduce the required storage overhead, are more efficient than previously proposed techniques and offer significant flexibility with regard to the input parameters. Experiments performed on real-life and synthetic data sets verify our theoretical study. •We prove a linear space lower bound.•A novel continuous algorithm is presented, which has two versions (COD).•To support different views of outliers, we propose an extension (ACOD).•We also propose algorithms based on micro-clusters (MCOD/AMCOD).•Performance evaluation results based on both real-life and synthetic data.