Fast parallel outlier detection for categorical datasets using MapReduce
Outlier detection has received considerable attention in many applications, such as detecting network attacks or credit card fraud The massive datasets currently available for mining in some of these outlier detection applications require large parallel systems, and consequently parallelizable outli...
Saved in:
Published in | 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) Vol. 10; pp. 3298 - 3304 |
---|---|
Main Authors | , , , , |
Format | Conference Proceeding Journal Article |
Language | English |
Published |
IEEE
01.06.2008
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Outlier detection has received considerable attention in many applications, such as detecting network attacks or credit card fraud The massive datasets currently available for mining in some of these outlier detection applications require large parallel systems, and consequently parallelizable outlier detection methods. Most existing outlier detection methods assume that all of the attributes of a dataset are numerical, usually have a quadratic time complexity with respect to the number of points in the dataset, and quite often they require multiple dataset scans. In this paper, we propose a fast parallel outlier detection strategy based on the Attribute Value Frequency (AVF) approach, a high-speed, scalable outlier detection method for categorical data that is inherently easy to parallelize. Our proposed solution, MR-AVF, is based on the MapReduce paradigm for parallel programming, which offers load balancing and fault tolerance. MR-AVF is particularly simple to develop and it is shown to be highly scalable with respect to the number of cluster nodes. |
---|---|
Bibliography: | ObjectType-Article-2 SourceType-Scholarly Journals-1 ObjectType-Feature-1 content type line 23 |
ISBN: | 1424418208 9781424418206 9781424432196 1424432197 |
ISSN: | 2161-4393 1522-4899 2161-4407 |
DOI: | 10.1109/IJCNN.2008.4634266 |