Fast and Exact Nearest Neighbor Search in Hamming Space on Full-Text Search Engines

• Published in: Similarity Search and Applications pp. 49 - 56
• Main Authors: Mu, Cun (Matthew), Zhao, Jun (Raymond), Yang, Guang, Yang, Binwei, Yan, Zheng (John)
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Elasticsearch; Full-text search engine; Hamming space; Lucene; Nearest neighbor search; Semantic binary embedding
• ISBN: 3030320464; 9783030320461
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-32047-8_5

A growing interest has been witnessed recently from both academia and industry in building nearest neighbor search (NNS) solutions on top of full-text search engines. Compared with other NNS systems, such solutions are capable of effectively reducing main memory consumption, coherently supporting multi-model search and being immediately ready for production deployment. In this paper, we continue the journey to explore specifically how to empower full-text search engines with fast and exact NNS in Hamming space (i.e., the set of binary codes). By revisiting three techniques (bit operation, subs-code filtering and data preprocessing with permutation) in information retrieval literature, we develop a novel engineering solution for full-text search engines to efficiently accomplish this special but important NNS task. In the experiment, we show that our proposed approach enables full-text search engines to achieve significant speed-ups over its state-of-the-art term match approach for NNS within binary codes.