Distributed Adaptive Binary Quantization for Fast Nearest Neighbor Search

• Published in: IEEE transactions on image processing Vol. 26; no. 11; pp. 5324 - 5336
• Main Authors: Liu, Xianglong, Li, Zhujin, Deng, Cheng, Tao, Dacheng
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.11.2017
• Subjects: Binary codes; binary quantization; distributed learning; Encoding; Hypercubes; Locality-sensitive hashing; nearest neighbor search; Nearest neighbor searches; product quantization; Prototypes; Quantization (signal); Training
• DOI: 10.1109/TIP.2017.2729896
• ISSN: 1057-7149; 1941-0042

Hashing has been proved an attractive technique for fast nearest neighbor search over big data. Compared with the projection based hashing methods, prototype-based ones own stronger power to generate discriminative binary codes for the data with complex intrinsic structure. However, existing prototype-based methods, such as spherical hashing and K-means hashing, still suffer from the ineffective coding that utilizes the complete binary codes in a hypercube. To address this problem, we propose an adaptive binary quantization (ABQ) method that learns a discriminative hash function with prototypes associated with small unique binary codes. Our alternating optimization adaptively discovers the prototype set and the code set of a varying size in an efficient way, which together robustly approximate the data relations. Our method can be naturally generalized to the product space for long hash codes, and enjoys the fast training linear to the number of the training data. We further devise a distributed framework for the large-scale learning, which can significantly speed up the training of ABQ in the distributed environment that has been widely deployed in many areas nowadays. The extensive experiments on four large-scale (up to 80 million) data sets demonstrate that our method significantly outperforms state-of-the-art hashing methods, with up to 58.84% performance gains relatively.