Imputing missing value through ensemble concept based on statistical measures

• Published in: Knowledge and information systems Vol. 56; no. 1; pp. 123 - 139
• Main Authors: Jenghara, Moslem Mohammadi, Ebrahimpour-Komleh, Hossein, Rezaie, Vahideh, Nejatian, Samad, Parvin, Hamid, Yusof, Sharifah Kamilah Syed
• Format: Journal Article
• Language: English
• Published: London Springer London 01.07.2018; Springer Nature B.V
• Subjects: Computer Science; Data mining; Data Mining and Knowledge Discovery; Database Management; Estimators; Information Storage and Retrieval; Information systems; Information Systems and Communication Service; Information Systems Applications (incl.Internet); IT in Business; Kernel functions; Kurtosis; Preprocessing; Regular Paper; Skewness; Statistical analysis; Statistical methods; Value management; Variance; Imputation; Kurtosis; Statistical distribution structure; Ensemble; Skewness; Missing value management
• ISSN: 0219-1377; 0219-3116
• DOI: 10.1007/s10115-017-1118-1

Many datasets include missing values in their attributes. Data mining techniques are not applicable in the presence of missing values. So an important step in preprocessing of a data mining task is missing value management. One of the most important categories in missing value management techniques is missing value imputation. This paper presents a new imputation technique. The proposed imputation technique is based on statistical measurements. The suggested imputation technique employs an ensemble of the estimators built to estimate the missing values based on positive and negative correlated observed attributes separately. Each estimator guesses a value for a missed value based on the average and variance of that feature. The average and variance of the feature are estimated from the non-missed values of that feature. The final consensus value for a missed value is the weighted aggregation of the values estimated by different estimators. The chief weight is attribute correlation, and the slight weight is dependent to kernel function such as kurtosis, skewness, number of involved samples and composition of them. The missing values are deliberately produced randomly at different levels. The experimentations indicate that the suggested technique has a good accuracy in comparison with the classical methods.