Conflicts to Harmony: A Framework for Resolving Conflicts in Heterogeneous Data by Truth Discovery

• Published in: IEEE transactions on knowledge and data engineering Vol. 28; no. 8; pp. 1986 - 1999
• Main Authors: Li, Yaliang, Li, Qi, Gao, Jing, Su, Lu, Zhao, Bo, Fan, Wei, Han, Jiawei
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.08.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Climatology; Computational efficiency; Computational modeling; Conflict resolution; Data fusion; Data models; Descriptions; Deviation; Estimation; heterogeneous data; Mathematical analysis; Mathematical models; Meteorology; Optimization; Product development; Raw materials; Reliability; truth discovery; Weather
• DOI: 10.1109/TKDE.2016.2559481
• ISSN: 1041-4347

In many applications, one can obtain descriptions about the same objects or events from a variety of sources. As a result, this will inevitably lead to data or information conflicts. One important problem is to identify the true information (i.e., the truths) among conflicting sources of data. It is intuitive to trust reliable sources more when deriving the truths, but it is usually unknown which one is more reliable a priori. Moreover, each source possesses a variety of properties with different data types. An accurate estimation of source reliability has to be made by modeling multiple properties in a unified model. Existing conflict resolution work either does not conduct source reliability estimation, or models multiple properties separately. In this paper, we propose to resolve conflicts among multiple sources of heterogeneous data types. We model the problem using an optimization framework where truths and source reliability are defined as two sets of unknown variables. The objective is to minimize the overall weighted deviation between the truths and the multi-source observations where each source is weighted by its reliability. Different loss functions can be incorporated into this framework to recognize the characteristics of various data types, and efficient computation approaches are developed. The proposed framework is further adapted to deal with streaming data in an incremental fashion and large-scale data in MapReduce model. Experiments on real-world weather, stock, and flight data as well as simulated multi-source data demonstrate the advantage of jointly modeling different data types in the proposed framework.