Automated anomaly detection service on heterogeneous log streams

• Main Authors: Zhang Hui, Jiang Guofei, Arora Nipun, Xu Jianwu, Debnath Biplob
• Format: Patent
• Language: English
• Published: 27.03.2018
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.