Memory Analysis Based Estimation of Hook Point by Virtual Machine Monitor

• Published in: International Journal of Networking and Computing Vol. 13; no. 2; pp. 273 - 286
• Main Authors: Sato, Masaya, Omori, Taku, Yamauchi, Toshihiro, Taniguchi, Hideo
• Format: Journal Article
• Language: English
• Published: IJNC Editorial Committee 2023
• Subjects: Operating System; System Call; Virtual Machine Monitor
• ISSN: 2185-2839; 2185-2847
• DOI: 10.15803/ijnc.13.2_273

The behavior of virtual machine (VM) programs are monitored by virtual machine monitors (VMMs) for security purposes. System calls are frequently used as a monitoring point. To monitor the system calls, the VMM inserts a breakpoint, called a hook point, into the memory of the monitored VM. The hook points are determined based on experimental knowledge. However, reading the source codes of operating systems (OSes) requires specialized knowledge. In addition, the appropriate hook point differs among OSes and OS versions. Analyzing the source code in each OS update is impractical. Searching for the appropriate hook point for various OSes is also difficult. To address these problems, we propose a method for estimating the hook point using a memory analysis technique. The proposed method acquires the memory of the monitored VM and then searches for an appropriate instruction appropriate to hook. The search instructions depend on the processor architecture. In addition, we also proposed a method for searching the appropriate instruction using a single step execution. This version reduces the cost for searching the instructions and improve robustness for various Linux versions. The experimental results showed that the proposed method precisely estimates the hook point for various OS versions and OSes. In addition, the overhead of the proposed method is small, considering the boot time of the monitored VM.