Unsupervised anomaly detection with LSTM autoencoders using statistical data-filtering

• Published in: Applied soft computing Vol. 108; p. 107443
• Main Authors: Maleki, Sepehr, Maleki, Sasan, Jennings, Nicholas R.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.09.2021
• Subjects: Anomaly detection; Autoencoders; Changepoint detection; IoT; LSTM networks; Changepoint detection; Autoencoders; LSTM networks; Anomaly detection; IoT
• ISSN: 1568-4946; 1872-9681
• DOI: 10.1016/j.asoc.2021.107443

To address one of the most challenging industry problems, we develop an enhanced training algorithm for anomaly detection in unlabelled sequential data such as time-series. We propose the outputs of a well-designed system are drawn from an unknown probability distribution, U, in normal conditions. We introduce a probability criterion based on the classical central limit theorem that allows evaluation of the likelihood that a data-point is drawn from U. This enables the labelling of the data on the fly. Non-anomalous data is passed to train a deep Long Short-Term Memory (LSTM) autoencoder that distinguishes anomalies when the reconstruction error exceeds a threshold. To illustrate our algorithm’s efficacy, we consider two real industrial case studies where gradually-developing and abrupt anomalies occur. Moreover, we compare our algorithm’s performance with four of the recent and widely used algorithms in the domain. We show that our algorithm achieves considerably better results in that it timely detects anomalies while others either miss or lag in doing so. •Anomaly detection in unlabelled Big Data is difficult and costly.•Distinguishing between true anomalies and expected changes is a challenge.•Autoencoders’ performance decline when data contains a lot of expected changes.•We introduce Enhanced LSTM AutoEncoders (ELSTMAE) for unsupervised anomaly detection in Big Data.