Analyst intuition inspired neural network based cyber security anomaly detection

Internet revolution has brought advancement to the world’s economy, business, technology and communication. It also brings forth the risk of cyber-attack penetration. It is a challenge to detect cyber-attack accurately and timely. In this work, we adopted a large network dataset containing malware attack data and trained it to recognize a cyber security attack to establish an expert system. The characteristics of attacker’s IP addresses can be extracted from our integrated dataset for statistical data extraction. The cyber security expert annotates the weight of each attribute and constructs a scoring system through log history annotation. We adopted a special semi supervise method to classify cyber security log into attack, unsure and no attack by first splitting the data into 3 clusters using fuzzy K-means (FKM), then manually label a small data (analyst intuition) and finally train the neural network classifier multi-layer perceptron (MLP) based on the manually labelled data. By doing so, our results were more encouraging as compared to finding anomalies within cyber security logs without analyst intuition’s labelling. The latter generally creates a large amount of false detections.