Utility Metrics for Evaluating Synthetic Health Data Generation Methods: Validation Study

• Published in: JMIR medical informatics Vol. 10; no. 4; p. e35734
• Main Authors: El Emam, Khaled, Mosquera, Lucy, Fang, Xi, El-Hussuna, Alaa
• Format: Journal Article
• Language: English
• Published: Canada JMIR Publications 07.04.2022
• Subjects: Cluster analysis; Datasets; Decision making; Multimedia; Original Paper; Privacy; Time series; Validation studies; Workloads; model validation; medical informatics; prediction model; synthetic data generation; utility metric; synthetic data; generative models; data privacy; data utility; binary prediction model; logistic regression
• DOI: 10.2196/35734
• ISSN: 2291-9694

A regular task by developers and users of synthetic data generation (SDG) methods is to evaluate and compare the utility of these methods. Multiple utility metrics have been proposed and used to evaluate synthetic data. However, they have not been validated in general or for comparing SDG methods. This study evaluates the ability of common utility metrics to rank SDG methods according to performance on a specific analytic workload. The workload of interest is the use of synthetic data for logistic regression prediction models, which is a very frequent workload in health research. We evaluated 6 utility metrics on 30 different health data sets and 3 different SDG methods (a Bayesian network, a Generative Adversarial Network, and sequential tree synthesis). These metrics were computed by averaging across 20 synthetic data sets from the same generative model. The metrics were then tested on their ability to rank the SDG methods based on prediction performance. Prediction performance was defined as the difference between each of the area under the receiver operating characteristic curve and area under the precision-recall curve values on synthetic data logistic regression prediction models versus real data models. The utility metric best able to rank SDG methods was the multivariate Hellinger distance based on a Gaussian copula representation of real and synthetic joint distributions. This study has validated a generative model utility metric, the multivariate Hellinger distance, which can be used to reliably rank competing SDG methods on the same data set. The Hellinger distance metric can be used to evaluate and compare alternate SDG methods.