Distributed Entropy Minimization Discretizer for Big Data Analysis under Apache Spark

The astonishing rate of data generation on the Internet nowadays has caused that many classical knowledge extraction techniques have become obsolete. Data reduction techniques are required in order to reduce the complexity order held by these techniques. Among reduction techniques, discretization is...

Full description

Saved in:
Bibliographic Details
Published in2015 IEEE Trustcom/BigDataSE/ISPA Vol. 2; pp. 33 - 40
Main Authors Ramirez-Gallego, Sergio, Garcia, Salvador, Mourino-Talin, Hector, Martinez-Rego, David, Bolon-Canedo, Veronica, Alonso-Betanzos, Amparo, Benitez, Jose Manuel, Herrera, Francisco
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.08.2015
Subjects
Big data
Complexity theory
Data mining
Distributed databases
Entropy
Minimization
Sparks
Online AccessGet full text
DOI10.1109/Trustcom.2015.559

Cover

More Information
Summary:The astonishing rate of data generation on the Internet nowadays has caused that many classical knowledge extraction techniques have become obsolete. Data reduction techniques are required in order to reduce the complexity order held by these techniques. Among reduction techniques, discretization is one of the most important tasks in data mining process, aimed at simplifying and reducing continuous-valued data in large datasets. In spite of the great interest in this reduction mechanism, only a few simple discretization techniques have been implemented in the literature for Big Data. Thereby we propose a distributed implementation of the entropy minimization discretizer proposed by Fayyad and Irani using Apache Spark platform. Our solution goes beyond a simple parallelization, transforming the iterativity yielded by the original proposal in a single-step computation. Experimental results on two large-scale datasets show that our solution is able to improve the classification accuracy as well as boosting the underlying learning process.
DOI:10.1109/Trustcom.2015.559