Leveraging malicious behavior traces from volatile memory using machine learning methods for trusted unknown malware detection in Linux cloud environments

Tomer Panker, Nir Nissim

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


Most organizations today use cloud-computing environments and virtualization technology. Linux-based clouds are the most popular cloud environments among organizations, and thus have become the target of cyber-attacks launched by sophisticated malware. Existing malware detection solutions for Linux-based VMs are installed and operated on the VM itself and are considered untrusted since malware can detect, interfere with, and even evade them. Thus, Linux cloud-based environments remain exposed to various malware-based attacks. This paper presents the first trusted framework for detecting unknown malware in Linux VM cloud-environments. Our framework acquires volatile memory dumps from the inspected VM by querying the hypervisor in a trusted manner and overcoming malware's ability to detect the security mechanism and evade detection. Then, using machine-learning algorithms we leverage informative traces (our 171 proposed features) from different parts of the VM's volatile memory. The framework was evaluated in seven rigorous experiments, on a total of 21,800 volatile memory dumps taken from two widely used virtual servers (10,900 from each server) during the execution of a diverse yet representative collection of benign and malicious Linux applications. Notably, the results show that our proposed framework can accurately (with high TPRs and low FPRs): (a) detect unknown malware (b) detect new unknown malware from unseen malware categories, which is a critical ability for coping with new malware trends and phenomena; (c) categorize an unknown malware by its attack category; (d) detect unknown malware on an unknown virtual-server; and lastly (e) detect fileless malware, a critical capability demonstrating the ability to detect substantially different attack modus operandi.

Original languageEnglish
Article number107095
JournalKnowledge-Based Systems
StatePublished - 17 Aug 2021


  • Cloud
  • Detection
  • Feature extraction
  • Linux
  • Machine learning
  • Malware
  • Virtual machine
  • Volatile memory
  • Volatility

ASJC Scopus subject areas

  • Management Information Systems
  • Software
  • Information Systems and Management
  • Artificial Intelligence


Dive into the research topics of 'Leveraging malicious behavior traces from volatile memory using machine learning methods for trusted unknown malware detection in Linux cloud environments'. Together they form a unique fingerprint.

Cite this