Sample-based quality estimation of query results in relational database environments

• Published in: IEEE transactions on knowledge and data engineering Vol. 18; no. 5; pp. 639 - 650
• Main Authors: Ballou, D.P., Chengalur-Smith, I.N., Wang, R.Y.
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.05.2006; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algebra; Applied sciences; Computer science; control theory; systems; Costs; Data processing. List processing. Character string processing; Data quality; database sampling; Estimates; Exact sciences and technology; information product; Information systems; Information systems. Data bases; Memory organisation. Data processing; Quality assessment; Quality control; Quality management; Query processing; relational algebra; Relational data bases; Relational databases; Samples; Sampling; Sampling methods; Software; Statistical analysis; Statistical methods; Studies; Tables; Warehousing; Yield estimation; Relational algebra; Statistical analysis; Probabilistic approach; Database query; Internet protocol; Relational database; Information retrieval; Data quality; Information use; Quality control; Database; Sampling; information product; database sampling
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2006.83

The quality of data in relational databases is often uncertain, and the relationship between the quality of the underlying base tables and the set of potential query results, a type of information product (IP), that could be produced from them has not been fully investigated. This paper provides a basis for the systematic analysis of the quality of such IPs. This research uses the relational algebra framework to develop estimates for the quality of query results based on the quality estimates of samples taken from the base tables. Our procedure requires an initial sample from the base tables; these samples are then used for all possible information IPs. Each specific query governs the quality assessment of the relevant samples. By using the same sample repeatedly, our approach is relatively cost effective. We introduce the reference-table procedure, which can be used for quality estimation in general. In addition, for each of the basic algebraic operators, we discuss simpler procedures that may be applicable. Special attention is devoted to the join operation. We examine various, relevant statistical issues, including how to deal with the impact on quality of missing rows in base tables. Finally, we address several implementation issues related to sampling.