MongoDB log collection and analysis system based on Kafka message queue

• Main Author: JIAO YUQIAN
• Format: Patent
• Language: Chinese; English
• Published: 18.01.2019
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a MongoDB log collection and analysis system based on Kafka message queue. The system includes: Kafka message queue log collection; consumer log parsing based on Java; log datastorage based on MongoDB; visual log analysis system; Compared with the prior art, the invention solves the load influence of high concurrent logging on the system; improves the efficiency of log information query; solves the problem of log information collection difficulty of high concurrency, high responsiveness IT application system; Improves the efficiency of log analysis, avoids parsing a large number of log files; cleans the data regularly to keep the data up to date. 本发明公开了基于Kafka消息队列的MongoDB日志采集和分析系统，该系统包括5部分：IT自研应用系统日志异步记录；Kafka消息队列日志收集；基于Java的消费者日志解析；基于MongoDB的日志数据存储；可视化日志分析系统；与现有技术相比，解决了高并发的日志记录对系统的负载影响；提高日志信息查询效率；解决高并发、高响应要求的IT应用系统的日志信息收集困难的问题；提高日志分析效率，避免去解析大量的日志文件；定期对数据进行清理，保证数据最新。