Data acquisition system, method and device based on message-oriented middleware

• Main Authors: ZHANG YULIANG, REN LINA, WANG PING, TU HONGWEI, NIE XIN, PENG JIAYONG, CHEN YE
• Format: Patent
• Language: Chinese; English
• Published: 02.04.2021
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a data acquisition system based on message-oriented middleware. The system comprises a main node, the message-oriented middleware, a plurality of child nodes and a database, and is characterized in that the main node is used for acquiring a data source URL list and distributing URLs in the data source URL list to the child nodes through the message-oriented middleware; andthe child nodes are realized based on a Scrapy acquisition framework and are used for acquiring data according to the received URLs and storing data acquisition results into a database through a database connection pool. Visibly, the system realizes an efficient data acquisition scheme in a distributed environment by expanding the Scrapy acquisition framework, and each node and the message middleware are relatively high in stability, so that the reliability of the whole data acquisition system can be ensured. In addition, the invention further provides a data acquisition method and device based on the message-oriented