A Ranking-Based Hashing Algorithm Based on the Distributed Spark Platform

• Published in: Information (Basel) Vol. 11; no. 3; p. 148
• Main Authors: Yang, Anbang, Qian, Jiangbo, Chen, Huahui, Dong, Yihong
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.03.2020
• Subjects: Algorithms; Big Data; Datasets; distributed framework; Efficiency; Hash based algorithms; Machine learning; Principal components analysis; Ranking; ranking-based hashing; Ratings & rankings; Semantics; similarity search; spark; Supervision; Teaching methods; Training
• ISSN: 2078-2489; 2078-2489
• DOI: 10.3390/info11030148

With the rapid development of modern society, generated data has increased exponentially. Finding required data from this huge data pool is an urgent problem that needs to be solved. Hashing technology is widely used in similarity searches of large-scale data. Among them, the ranking-based hashing algorithm has been widely studied due to its accuracy and speed regarding the search results. At present, most ranking-based hashing algorithms construct loss functions by comparing the rank consistency of data in Euclidean and Hamming spaces. However, most of them have high time complexity and long training times, meaning they cannot meet requirements. In order to solve these problems, this paper introduces a distributed Spark framework and implements the ranking-based hashing algorithm in a parallel environment on multiple machines. The experimental results show that the Spark-RLSH (Ranking Listwise Supervision Hashing) can greatly reduce the training time and improve the training efficiency compared with other ranking-based hashing algorithms.