Website data automatic crawling method based on JMeter

The invention relates to the field of web, in particular to an automatic website data crawling method based on JMeter, which avoids a large number of complex JS operations involved in the process of crawling data from a front-end interface. Meanwhile, the long data crawling process or direct crawlin...

Full description

Saved in:
Bibliographic Details
Main Authors TANG JUN, YANG XUEMEI, LIU CHUXIONG
Format Patent
LanguageChinese
English
Published 22.01.2021
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:The invention relates to the field of web, in particular to an automatic website data crawling method based on JMeter, which avoids a large number of complex JS operations involved in the process of crawling data from a front-end interface. Meanwhile, the long data crawling process or direct crawling failure caused by the limitation of certain websites on the HTTP request frequency and access frequency within a certain period of time is also avoided. According to the technical scheme, the method comprises the steps that a target website needing data crawling is determined, then data analysis is conducted on the target website, a data interface and attribute information corresponding to the data interface are obtained, the data interface is executed at a JMeter end, and whether request parameters in the data interface and corresponding response results meet expected settings or not is checked; if the exception is met, parameter dynamic configuration is conducted on the data interface, dynamic configuration is
Bibliography:Application Number: CN202011156240