Real-Time Distribution Algorithm for Fully Comparison Data Based on Storm

• Published in: Mobile networks and applications Vol. 27; no. 2; pp. 588 - 597
• Main Authors: Dong, Chang-qing, Chen, Chen, Ren, Nver, Cai, Jian-jun
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.04.2022; Springer Nature B.V
• Subjects: Algorithms; Clustering; Communications Engineering; Computer Communication Networks; Electrical Engineering; Engineering; Graph theory; IT in Business; Networks; Phase unwrapping; Read only; Real time; Redundancy; Transaction processing; Real time; Storm framework; Full comparison data; Data allocation
• ISSN: 1383-469X; 1572-8153
• DOI: 10.1007/s11036-021-01824-3

Current data allocation algorithms neglect the problems of unsatisfactory allocation results and long execution time caused by the redundancy of full comparative data and the complexity of data types. To solve these problems, a real-time allocation algorithm of full comparison data based on storm is proposed. Firstly, the phase unwrapping algorithm of minimum spanning tree is used to remove redundant data in full comparison data; then, the distributed data clustering algorithm and storm framework are used to realize the full comparison data clustering after redundancy removal. Several main factors affecting the selection of statistical information are summarized according to the clustering results. Then the communication cost of data loading and transaction processing is determined, and the trade-off between read-only transaction and update transaction cost is achieved. By judging whether the total cost of read-only transaction and update transaction is reduced or not, the replica is eliminated, and a full comparison data allocation algorithm with minimum total cost of read-only transaction and update transaction is proposed to realize real-time allocation of full-comparative data. The example analysis shows that the proposed algorithm can meet the user’s needs in terms of execution time, acceleration ratio, storage efficiency and cost. Compared with the reference algorithm, the proposed algorithm has the lowest execution time, the highest acceleration ratio and the closest allocation cost to the ideal overhead.