AMiner: A Modular Log Data Analysis Pipeline for Anomaly-based Intrusion Detection

Cyber attacks are omnipresent and their rapid detection is crucial for system security. Signature-based intrusion detection monitors systems for attack indicators and plays an important role in recognizing and preventing such attacks. Unfortunately, it is unable to detect new attack vectors and may be evaded by attack variants. As a solution, anomaly detection employs techniques from machine learning to detect suspicious log events without relying on predefined signatures. While visibility of attacks in network traffic is limited due to encryption of network packets, system log data is available in raw format and thus allows fine-granular analysis. However, system log processing is difficult as it involves different formats and heterogeneous events. To ease log-based anomaly detection, we present the AMiner, an open-source tool in the AECID toolbox that enables fast log parsing, analysis, and alerting. In this article, we outline the AMiner’s modular architecture and demonstrate its applicability in three use-cases.