Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

• Published in: IEEE access Vol. 12; pp. 78193 - 78218
• Main Authors: Lupton, Scott, Washizaki, Hironori, Yoshioka, Nobukazu, Fukazawa, Yoshiaki
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2024; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithms; Anomalies; Anomaly detection; Bibliographies; Complex systems; Deep learning; Log parsing; log template extraction; Measurement; Monitoring; online algorithms; Online services; Parsers; Systematics; Taxonomy
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3387287

As production system estates become larger and more complex, ensuring stability through traditional monitoring approaches becomes more challenging. Rule-based monitoring is common in industrial settings, but it has limitations. These include the difficulty of crafting rules capable of detecting unforeseen issues and the burden of manually maintaining rule sets. A potential solution to effectively manage complex system states is log anomaly detection. Workflows for log anomaly detection utilize several fundamental components. These include preprocessors for data cleansing, parsers to extract structured information from raw log data, encoding algorithms to convert extracted data into usable model input features, anomaly detection methods to isolate anomalous signals, and feedback mechanisms to incrementally improve model performance. This study explores the current state of research into online parser-supported log anomaly detection methods, investigates recent research trends, compares the performances of parser and anomaly detection methods using common public datasets and metrics, and assesses their performance evolution over time. Additionally, it classifies available methods using a newly introduced taxonomy, highlights current research gaps, and recommends future research directions.