Multiple outlier detection in multivariate data using self-organizing maps title

• Published in: Computational statistics Vol. 20; no. 2; pp. 245 - 264
• Main Authors: Nag, Ashok K., Mitra, Amit, Mitra, Sharmishtha
• Format: Journal Article
• Language: English
• Published: Heidelberg Springer Nature B.V 01.06.2005
• Subjects: Applied statistics; Artificial intelligence; Cheques; Communities; Computational mathematics; Datasets; Expert systems; Multidimensional data; Statistics; Studies
• ISSN: 0943-4062; 1613-9658
• DOI: 10.1007/BF02789702

The problem of detection of multidimensional outliers is a fundamental and important problem in applied statistics. The unreliability of multivariate outlier detection techniques such as Mahalanobis distance and hat matrix leverage has led to development of techniques which have been known in the statistical community for well over a decade. The literature on this subject is vast and growing. In this paper, we propose to use the artificial intelligence technique ofself-organizing map (SOM) for detecting multiple outliers in multidimensional datasets. SOM, which produces a topology-preserving mapping of the multidimensional data cloud onto lower dimensional visualizable plane, provides an easy way of detection of multidimensional outliers in the data, at respective levels of leverage. The proposed SOM based method for outlier detection not only identifies the multidimensional outliers, it actually provides information about the entire outlier neighbourhood. Being an artificial intelligence technique, SOM based outlier detection technique is non-parametric and can be used to detect outliers from very large multidimensional datasets. The method is applied to detect outliers from varied types of simulated multivariate datasets, a benchmark dataset and also to real life cheque processing dataset. The results show that SOM can effectively be used as a useful technique for multidimensional outlier detection.[PUBLICATION ABSTRACT]