Automatic detection of multi-line templates in software log files

• Published in: 2017 Seventeenth International Conference on Advances in ICT for Emerging Regions (ICTer) pp. 1 - 8
• Main Authors: Jayathilake, P. W. D. C., Weeraddana, N. R., Hettiarachchi, H. K. E. P.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.09.2017
• Subjects: Arrays; Automation; Data Extraction; Data mining; Knowledge Representation; Log analysis; Particle separators; Pattern Detection; Periodic structures; Regular Expressions; Software; Suffix Tree
• ISBN: 9781538624432; 1538624435
• ISSN: 2472-7598
• DOI: 10.1109/ICTER.2017.8257824

This research proposes a method for automatically identifying and coding the structure in any text log file. Generated code is expressed in the declarative language LDEL, which was designed and implemented during earlier stages of this research. Once the code is available, the LDEL parser is able to scan a log file with the particular structure and extract log information into a tree-based data structure. Subsequently the extracted information can be processed according to prespecified logic, which is expressed in a procedural language which also was developed as a prior stage in this research. The language scheme used for expressing log file structure was carefully designed after examining log files from numerous domains as a result of the main author's previous research. This scheme is capable of capturing peculiar patterns occurring both within and across lines in log files. Since the development of this scheme and the procedural language, it was noticed that significant manual work was needed for a human expert to examine the log file and write the script that captures its structure. This approach was both time consuming and error prone. With this research we propose and evaluate a method to automatically detect a larger part of the log file structure so that the human specialist can use the generated script with minimal or no modifications.