Data processing method and system for big data

• Main Authors: ZHI TING, QI GANG, CAO YANG, HUANG HAIFENG, CHEN LUN, LI BAODONG, HAN GUOQUAN
• Format: Patent
• Language: Chinese; English
• Published: 01.11.2022
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC COMMUNICATION TECHNIQUE; ELECTRIC DIGITAL DATA PROCESSING; ELECTRICITY; PHYSICS; TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHICCOMMUNICATION

The invention relates to the field of information processing, and discloses a big data processing method and system.The method comprises the steps that a name node, a data node and a user side are arranged, the name node is configured to be a center management server, descriptive metadata are stored in a memory of the name node in a list mode, and the data node is configured to be a user side; providing an internal metadata service in response to an access request of a user side to a file; a static cache queue is arranged at the name node, the static cache queue stores block access data corresponding to a hot file, and a redirection message is set at the data node, so that data access and address change can be fed back to a user side in time, and the convenience of accessing data node information by a user is realized. 本发明涉及信息处理领域，公开了一种大数据的数据处理方法和系统，所述方法包括设置名称节点，数据节点和用户端，其中，所述名称节点被配置为中心管理服务器，将描述性元数据以列表形式存储在名称节点的内存中，响应于用户端对文件的访问请求，提供内部元数据服务；在所述名称节点处，设置有静态缓存队列，所述静态缓存队列，存储对应热点文件的分块访问数据，以及在数据节点设置重定向消息，将数据的访问和地址变更