iDMI: A novel technique for missing value imputation using a decision tree and expectation-maximization algorithm

• Published in: 16th Int'l Conf. Computer and Information Technology pp. 496 - 501
• Main Authors: Rahman, Md Geaur, Islam, Md Zahidul
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.03.2014
• Subjects: Accuracy; Computers; Correlation; data cleansing; Data pre-processing; Decision trees; Electromagnetic interference; EM algorithm; Information technology; missing value imputation; Remuneration
• DOI: 10.1109/ICCITechn.2014.6997351

In this paper we present a novel technique called iDMI that imputes missing values of a data set by combining a decision tree algorithm (DT) and an expectation-maximization (EMI) algorithm. We first divide a data set into horizontal segments through applying a DT algorithm such as C4.5, and then apply an EMI algorithm on each segment in order to impute the missing values belong to the segment. If all numerical attribute values of a record are missing then we impute them by the mean values of the attributes of the records belong to a segment where the record falls in, and thereby reduce the computational time complexity of iDMI compare to an existing technique called DMI which calculate the mean value of an attribute by using all records of a data set. We evaluate the performance of iDMI over three high quality existing techniques on two real data sets in terms of four evaluation criteria. Our initial experimental results, including several statistical significance analysis, indicate the superiority of iDMI over the existing techniques.