A Methodology for the Automatic Generation of Realistic Log Datasets of a Windows Domain for Testing Cybersecurity Machine Learning Algorithms

Intrusion detection systems are among the essential security assets in enterprise environments, providing multiple layers of security and aligning with the principles of defense in depth [1]. Their strength, compared to different security appliances, lies in the detection based on anomalies achieved through machine learning algorithms, offering protection against known and unknown attacks, such as zero-day exploits, by monitoring system behavior and network traffic patterns [2]. However, the efficacy of IDS relies on the quality of the dataset used to train it, which often is outdated and does not reflect the current threat landscape [3],[4]. Especially in the area of Windows, there is a research and dataset gap [5]. This thesis addresses this gap by proposing the AECID-Win-Testbed framework, aiming for the automated generation of log datasets specific to Windows environments, built upon the principles of model-driven engineering. Requirements were formulated for Windows log datasets, upon which the framework design was based. In an iterative development process, the initial iteration focused on evaluating components necessary for deploying a security-focused testbed. This included log management, downstream processing of logs, user simulation and the tooling for automated deployment. As the focus is on Windows enterprises, domain-specific configurations were reviewed, including group policy objects and audit rules. Additionally, attacks were executed which cover a wide range of tactics and techniques from the MITRE ATT\&CK framework. In the second iteration, the technical implementation of the AECID-Win-Testbed is presented. This implementation followed the principles of MDE, defining models for the testbed, user simulation and attack chain. The integration of GHOSTS-NPC ensured realistic user simulations and Attackmate was utilized for the automated execution of an attack chain. A conceptual model of the layers, including model, testbed, collection and dataset, was presented. Functional testing of the framework was conducted and the resulting dataset was evaluated. This evaluation involved assigning attributes and calculating metrics such as precision, recall and F1-score. The results demonstrated that the dataset generated from the framework is usable and overall confirmed the functionality of the AECID-Win-Testbed itself. The research follows the proposed methodology of MDE and highlights its feasibility for the automated generation of datasets for IDS. Configurations, scripts and the dataset itself will be published by AIT simultaneously with this thesis.