Anomaly detection for individual sequences with applications in identifying malicious tools

Shachar Siboni, Asaf Cohen

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Anomaly detection refers to the problem of identifying abnormal behaviour within a set of measurements. In many cases, one has some statistical model for normal data, and wishes to identify whether new data fit the model or not. However, in others, while there are normal data to learn from, there is no statistical model for this data, and there is no structured parameter set to estimate. Thus, one is forced to assume an individual sequences setup, where there is no given model or any guarantee that such a model exists. In this work, we propose a universal anomaly detection algorithm for one-dimensional time series that is able to learn the normal behaviour of systems and alert for abnormalities, without assuming anything on the normal data, or anything on the anomalies. The suggested method utilizes new information measures that were derived from the Lempel-Ziv (LZ) compression algorithm in order to optimally and efficiently learn the normal behaviour (during learning), and then estimate the likelihood of new data (during operation) and classify it accordingly. We apply the algorithm to key problems in computer security, as well as a benchmark anomaly detection data set, all using simple, single-feature time-indexed data. The first is detecting Botnets Command and Control (C&C) channels without deep inspection. We then apply it to the problems of malicious tools detection via system calls monitoring and data leakage identification. We conclude with the New York City (NYC) taxi data. Finally, while using information theoretic tools, we show that an attacker's attempt to maliciously fool the detection system by trying to generate normal data is bound to fail, either due to a high probability of error or because of the need for huge amounts of resources.

Original languageEnglish
Article number649
JournalEntropy
Volume22
Issue number6
DOIs
StatePublished - 1 Jun 2020

Keywords

  • Anomaly detection
  • Botnets
  • Command and control channels
  • Computer security
  • Individual sequences
  • Learning
  • NYC taxi data
  • One-dimensional time series
  • Probability assignment
  • Statistical model
  • Universal compression

ASJC Scopus subject areas

  • Physics and Astronomy (all)

Fingerprint

Dive into the research topics of 'Anomaly detection for individual sequences with applications in identifying malicious tools'. Together they form a unique fingerprint.

Cite this