A game theoretic-based distributed detection method for VM-to-hypervisor attacks in cloud environment

• Published in: The Journal of supercomputing Vol. 73; no. 10; pp. 4407 - 4427
• Main Authors: Nezarat, Amin, Shams, Yaser
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.10.2017; Springer Nature B.V
• Subjects: Agents (artificial intelligence); Cloud computing; Compilers; Computer Science; Computer simulation; Cybersecurity; Firewalls; Game theory; Interpreters; Intrusion detection systems; Processor Architectures; Programming Languages; Scheduling; Cloud computing; Nash equilibrium; Virtualization; Game theory; Intrusion detection systems
• DOI: 10.1007/s11227-017-2025-7
• ISSN: 0920-8542

Cloud computing is a pool of scalable virtual resources serving a large number of users who pay fees depending on the extent of utilized service. From payment perspective, cloud is like electricity and water as people who use more of this shared pool should pay larger fees. Cloud computing involves a diverse set of technologies including networking, virtualization and transaction scheduling. Thus, it is vulnerable to a wide range of security threats. Some of the most important security issues threatening the cloud computing systems originate from virtualization technology, as it constitutes the main body and basis of these systems. The most important virtualization-based security threats include VM side channel, VM escape and rootkit attacks. The previous works on the subject of virtualization security rely on hardware approaches such as the use of firewalls, which are expensive, the use of schedulers to control the side channels along with noise injection, which impose high overhead, or the use of agents to collect information and send them back to a central intrusion detection system, which itself can become the target of attacker. In the method presented in this paper, a group of mobile agents act as the sensors of invalid actions in the cloud environment. They start a noncooperative game with the suspected attacker and then calculate the Nash equilibrium value and utility so as to differentiate an attack from legitimate requests and determine the severity of attack and its point of origin. The simulation results show that this method can detect the attacks with 86% accuracy. The use of mobile agents and their trainability feature has led to reduced system overhead and accelerated detection process.