Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining

Nir Nissim, Yuval Lapidot, Aviad Cohen, Yuval Elovici

Research output: Contribution to journalArticlepeer-review

28 Scopus citations


Most organizations today employ cloud-computing environments and virtualization technology; Due to their prevalence and importance in providing services to the entire organization, virtual-servers are constantly targeted by cyber-attacks, and specifically by malware. Existing solutions, consisting of the widely-used antivirus (AV) software, fail to detect newly created and unknown-malware; moreover, by the time the AV is updated, the organization has already been attacked. In this paper, we present a during run-time analysis methodology for a trusted detection of unknown malware on virtual machines (VMs). We conducted trusted analysis of volatile memory dumps taken from a VM and focused on analyzing their system-calls using a sequential-mining-method. We leveraged the most informative system-calls by machine-learning algorithms for the efficient detection of malware in widely used VMs within organizations (i.e. IIS and Email server). We evaluated our methodology in a comprehensive set of experiments over a collections of real-world, advanced, and notorious malware (both ransomware and RAT), and legitimate programs. The results show that our suggested methodology is able to detect the presence of unknown malware, in an average of 97.9% TPR and 0% FPR. Such results and capabilities can form the ground for the development of practical detection-tools for both corporates and companies.

Original languageEnglish
Pages (from-to)147-175
Number of pages29
JournalKnowledge-Based Systems
StatePublished - 1 Aug 2018


  • Machine learning
  • Malware detection
  • Memory dump
  • Private cloud
  • Ransomware
  • Remote access Trojan
  • Sequential mining
  • Virtual machine
  • Virtual server
  • Volatile memory


Dive into the research topics of 'Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining'. Together they form a unique fingerprint.

Cite this