IPLog: An Efficient Log Parsing Method Based on Few-Shot Learning

• Published in: Electronics (Basel) Vol. 13; no. 16; p. 3324
• Main Authors: Liu, Shuxian, Yun, Libo, Nie, Shuaiqi, Zhang, Guiheng, Li, Wei
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.08.2024
• Subjects: Analysis; Anomalies; Automation; Clustering; Computational linguistics; Datasets; Frequency distribution; Language processing; Learning; Messages; Methods; Natural language interfaces; Pattern recognition; Semantics; Software; Structured data; Task complexity
• ISSN: 2079-9292; 2079-9292
• DOI: 10.3390/electronics13163324

Log messages from enterprise-level software systems contain crucial runtime details. Engineers can convert log messages into structured data through log parsing, laying the foundation for downstream tasks such as log anomaly detection. Existing log parsing schemes usually underperform in production environments for several reasons: first, they often ignore the semantics of log messages; second, they are often not adapted to different systems, and their performance varies greatly; and finally, they are difficult to adapt to the complexity and variety of log formats in the real environment. In response to the limitations of current approaches, we introduce IPLog (Intelligent Parse Log), a parsing method designed to address these issues. IPLog samples a limited set of log samples based on the distribution of templates in the system’s historical logs, and allows the model to make full use of the small number of log samples to recognize common patterns of keywords and parameters through few-shot learning, and thus can be easily adapted to different systems. In addition, IPLog can further improve the grouping accuracy of log templates through a novel manual feedback merge query strategy based on the longest common prefix, thus enhancing the model’s adaptability to handle complex log formats in production environments. We conducted experiments on four newly released public log datasets, and the experimental results show that IPLog can achieve an average grouping accuracy (GA) of 0.987 and parsing accuracy (PA) of 0.914 on the four public datasets, which are the best among the mainstream parsing schemes. These results demonstrate that IPLog is effective for log parsing tasks.