Simple multi-party set reconciliation

• Published in: Distributed computing Vol. 31; no. 6; pp. 441 - 453
• Main Authors: Mitzenmacher, Michael, Pagh, Rasmus
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.11.2018; Springer Nature B.V
• Subjects: Cloud computing; Coding; Computer Communication Networks; Computer Hardware; Computer Science; Computer Systems Organization and Communication Networks; Lookup tables; Reconciliation; Software Engineering/Programming and Operating Systems; Synchronism; Theory of Computation; Multi-party; Network coding; Hashing; Set reconciliation; Sketching
• ISSN: 0178-2770; 1432-0452
• DOI: 10.1007/s00446-017-0316-0

Many distributed cloud-based services use multiple loosely consistent replicas of user information to avoid the high overhead of more tightly coupled synchronization. Periodically, the information must be synchronized, or reconciled. One can place this problem in the theoretical framework of set reconciliation : two parties A 1 and A 2 each hold a set of keys, named S 1 and S 2 respectively, and the goal is for both parties to obtain S 1 ∪ S 2 . Typically, set reconciliation is interesting algorithmically when sets are large but the set difference | S 1 - S 2 | + | S 2 - S 1 | is small. In this setting the focus is on accomplishing reconciliation efficiently in terms of communication; ideally, the communication should depend on the size of the set difference, and not on the size of the sets. In this paper, we extend recent approaches using Invertible Bloom Lookup Tables (IBLTs) for set reconciliation to the multi-party setting. There are three or more parties A 1 , A 2 , … , A n holding sets of keys S 1 , S 2 , … , S n respectively, and the goal is for all parties to obtain ∪ i S i . While this could be done by pairwise reconciliations, we seek more effective methods. Our general approach can function even if the number of parties is not exactly known in advance, and with some additional cost can be used to determine which other parties hold missing keys. Our methodology uses network coding techniques in conjunction with IBLTs, allowing efficiency in network utilization along with efficiency obtained by passing messages of size O ( | ∪ i S i - ∩ i S i | ) . By connecting reconciliation with network coding, we can provide efficient reconciliation methods for a number of natural distributed settings.