PC-Filter: A Robust Filtering Technique for Duplicate Record Detection in Large Databases

In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts th...

Full description

Saved in:
Bibliographic Details
Published inDatabase and Expert Systems Applications pp. 486 - 496
Main Authors Zhang, Ji, Ling, Tok Wang, Bruckner, Robert M., Liu, Han
Format Book Chapter Conference Proceeding
LanguageEnglish
Published Berlin, Heidelberg Springer Berlin Heidelberg 2004
Springer
SeriesLecture Notes in Computer Science
Subjects
Applied sciences
Computer science; control theory; systems
Edit Distance
Exact sciences and technology
Information systems. Data bases
Longe Common Subsequence
Memory organisation. Data processing
Recall Level
Record Similarity
Software
Selection problem
Filtering
Paired comparison
Database
Data partition
Case based reasoning
Program specification
Online AccessGet full text

Cover

More Information
Summary:In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of O(N4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the ”Key Selection” problem, the ”Scope Specification” problem and the ”Low Recall” problem that the existing methods suffer from.
ISBN:9783540229360
3540229361
ISSN:0302-9743
1611-3349
DOI:10.1007/978-3-540-30075-5_47