Probabilistic outlier detection for sparse multivariate geotechnical site investigation data using Bayesian learning

• Published in: Di xue qian yuan. Vol. 12; no. 1; pp. 425 - 439
• Main Authors: Zheng, Shuo, Zhu, Yu-Xin, Li, Dian-Qing, Cao, Zi-Jun, Deng, Qin-Xuan, Phoon, Kok-Kwang
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.01.2021; State Key Laboratory of Water Resources and Hydropower Engineering Science, Institute of Engineering Risk and Disaster Prevention, Wuhan University, 299 Bayi Road,Wuhan 430072, China%Department of Civil and Environmental Engineering National University of Singapore, Blk E1A, #07-03, 1 Engineering Drive 2, Singapore 117576, Singapore
• Subjects: Bayesian machine learning; Mahalanobis distance; Outlier detection; Resampling by half-means; Site investigation; Sparse multivariate data; Resampling by half-means; Outlier detection; Sparse multivariate data; Bayesian machine learning; Mahalanobis distance; Site investigation
• ISSN: 1674-9871
• DOI: 10.1016/j.gsf.2020.03.017

Various uncertainties arising during acquisition process of geoscience data may result in anomalous data instances (i.e., outliers) that do not conform with the expected pattern of regular data instances. With sparse multivariate data obtained from geotechnical site investigation, it is impossible to identify outliers with certainty due to the distortion of statistics of geotechnical parameters caused by outliers and their associated statistical uncertainty resulted from data sparsity. This paper develops a probabilistic outlier detection method for sparse multivariate data obtained from geotechnical site investigation. The proposed approach quantifies the outlying probability of each data instance based on Mahalanobis distance and determines outliers as those data instances with outlying probabilities greater than 0.5. It tackles the distortion issue of statistics estimated from the dataset with outliers by a re-sampling technique and accounts, rationally, for the statistical uncertainty by Bayesian machine learning. Moreover, the proposed approach also suggests an exclusive method to determine outlying components of each outlier. The proposed approach is illustrated and verified using simulated and real-life dataset. It showed that the proposed approach properly identifies outliers among sparse multivariate data and their corresponding outlying components in a probabilistic manner. It can significantly reduce the masking effect (i.e., missing some actual outliers due to the distortion of statistics by the outliers and statistical uncertainty). It also found that outliers among sparse multivariate data instances affect significantly the construction of multivariate distribution of geotechnical parameters for uncertainty quantification. This emphasizes the necessity of data cleaning process (e.g., outlier detection) for uncertainty quantification based on geoscience data. [Display omitted] •A probabilistic method is proposed for detecting outliers from multivariate data.•Distortion in statistics of geo-parameters due to outliers is taken into account.•Statistical uncertainty due to sparsity of site-specific data is quantified by BML.•The proposed approach is illustrated and verified using simulated and real data.