Algorithms and performance of load-balancing with multiple hash functions in massive content distribution

• Published in: Computer networks (Amsterdam, Netherlands : 1999) Vol. 53; no. 1; pp. 110 - 125
• Main Authors: Xia, Ye, Chen, Shigang, Cho, Chunglae, Korgaonkar, Vivekanand
• Format: Journal Article
• Language: English
• Published: Kidlington Elsevier B.V 16.01.2009; Elsevier; Elsevier Sequoia S.A
• Subjects: Algorithms; Applied sciences; Business and industry local networks; Content distribution network; Distributed-hash-table; Exact sciences and technology; File-sharing; Infrastructure; Load-balancing; Miscellaneous; Networks and services in france and abroad; Peer to peer computing; Peer-to-peer network; Studies; Systems, networks and services of telecommunications; Telecommunications; Telecommunications and information theory; Teleprocessing networks. Isdn; Transmission and modulation (techniques and equipments); Valuation and optimization of characteristics. Simulation; Web content delivery; Distributed-hash-table; Peer-to-peer network; File-sharing; Load-balancing; Content distribution network; Performance evaluation; Peer to peer; Discrete Hartley transforms; Resource sharing; Information delivery; Failures; Implementation; Local network; Search algorithm; Random search; Algorithm performance; Computer network; Overload; Distributed algorithm; Hashing; Data transmission network; Replication; Load balancing; Distribution network
• ISSN: 1389-1286; 1872-7069
• DOI: 10.1016/j.comnet.2008.10.003

We consider a two-tier content distribution system for distributing massive content, consisting of an infrastructure content distribution network (CDN) and a large number of ordinary clients. The nodes of the infrastructure network form a structured, distributed-hash-table-based (DHT) peer-to-peer (P2P) network. Each file is first placed in the CDN, and possibly, is replicated among the infrastructure nodes depending on its popularity. In such a system, it is particularly pressing to have proper load-balancing mechanisms to relieve server or network overload. The subject of the paper is on popularity-based file replication techniques within the CDN using multiple hash functions. Our strategy is to set aside a large number of hash functions. When the demand for a file exceeds the overall capacity of the current servers, a previously unused hash function is used to obtain a new node ID where the file will be replicated. The central problems are how to choose an unused hash function when replicating a file and how to choose a used hash function when requesting the file. Our solution to the file replication problem is to choose the unused hash function with the smallest index, and our solution to the file request problem is to choose a used hash function uniformly at random. Our main contribution is that we have developed a set of distributed, robust algorithms to implement the above solutions and we have evaluated their performance. In particular, we have analyzed a random binary search algorithm for file request and a random gap removal algorithm for failure recovery.