Unsupervised, Near Real-Time, and Explainable Anomaly Detection on Network Data Using Multiple Instances of Micro-Cluster Detection

Abstract

With the increasing sophistication of cyber threats and the growing volume of data being transmitted over networks, ensuring the security and integrity of online systems has become a challenge. The analysis of network data logs has been a popular approach due to the ease of implementation of solutions given that log capturing is usually already put in place for tasks related to monitoring, debugging, and performance measuring, among others. Considering that logs describe the actions being performed on a system, detecting an attack is analogous to finding anomalous actions being performed. This research presents an unsupervised anomaly detection framework, MIMC, based on network and service log data that improves the current state-of-the-art. It does so by expanding on the number of attributes analyzed and determining the anomaly status of any given log by combining individually acquired results from each attribute. A comprehensive study on the impact of different configurations of the proposed method over different datasets is also presented. Obtained results show that MIMC yields higher performance than the comparable state-of-the-art method, MIDAS, in all evaluated scenarios and datasets. By employing a novel technique for determining system parameters, MIMC can considerably improve the yielded performance in most cases, which is shown to have statistical significance despite the non-deterministic nature of both methods.