Data processing method and device based on Hadoop cluster, equipment and storage medium

The invention discloses a data processing method, device and equipment based on Hadoop clusters and a storage medium, and the data processing method based on the Hadoop clusters comprises the following steps: deploying at least two Hadoop clusters based on the same software and hardware resources an...

Full description

Saved in:
Bibliographic Details
Main Authors XU JILAI, LUO XIAOFENG, LIN FAQUAN, DU TENGFEI, JIANG FAN, ZHANG YANTANG
Format Patent
LanguageChinese
English
Published 03.01.2023
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:The invention discloses a data processing method, device and equipment based on Hadoop clusters and a storage medium, and the data processing method based on the Hadoop clusters comprises the following steps: deploying at least two Hadoop clusters based on the same software and hardware resources and configuration rules; wherein the at least two Hadoop clusters are deployed in different places, and the distance between the two Hadoop clusters is greater than a distance threshold; synchronously completing data import and data processing on the at least two Hadoop clusters by using the scheduling platform; monitoring the operation states of the at least two Hadoop clusters, and determining available Hadoop clusters; and determining a target Hadoop cluster from the available Hadoop clusters to provide a data service for the data demand side. According to the technical scheme, the adaptive capacity of the Hadoop cluster to external environment changes is improved, and the stability of the Hadoop cluster for provi
Bibliography:Application Number: CN202211265436