LogLens: A Real-Time Log Analysis System

• Published in: 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) pp. 1052 - 1062
• Main Authors: Debnath, Biplob, Solaimani, Mohiuddin, Gulzar, Muhammad Ali Gulzar, Arora, Nipun, Lumezanu, Cristian, Xu, JianWu, Zong, Bo, Zhang, Hui, Jiang, Guofei, Khan, Latifur
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.07.2018
• Subjects: Adaptation models; Analytical models; Anomaly detection; Big Data Processing; Detectors; Distributed Computing; Event Driven Compution; Log Analysis; Log Parsing; Log Sequence Violation; Real time system; Real-time systems; Spark; Sparks; Unsupervised Learning
• ISSN: 2575-8411
• DOI: 10.1109/ICDCS.2018.00105

Administrators of most user-facing systems depend on periodic log data to get an idea of the health and status of production applications. Logs report information, which is crucial to diagnose the root cause of complex problems. In this paper, we present a real-time log analysis system called LogLens that automates the process of anomaly detection from logs with no (or minimal) target system knowledge and user specification. In LogLens, we employ unsupervised machine learning based techniques to discover patterns in application logs, and then leverage these patterns along with the real-time log parsing for designing advanced log analytics applications. Compared to the existing systems which are primarily limited to log indexing and search capabilities, LogLens presents an extensible system for supporting both stateless and stateful log analysis applications. Currently, LogLens is running at the core of a commercial log analysis solution handling millions of logs generated from the large-scale industrial environments and reported up to 12096x man-hours reduction in troubleshooting operational problems compared to the manual approach.