The Thundering Herd: Amplifying Kernel Interference to Attack Response Times

• Published in: Proceedings / IEEE Real-Time and Embedded Technology and Applications Symposium pp. 95 - 107
• Main Authors: Mergendahl, Samuel, Jero, Samuel, Ward, Bryan C., Furgala, Juliana, Parmer, Gabriel, Skowyra, Richard
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.05.2022
• Subjects: Instruction sets; Interference; kernel-interference; Malware; Mission critical systems; Real-time systems; seL4; Servers; temporal-isolation; Thundering-Herd; Time factors
• ISSN: 2642-7346
• DOI: 10.1109/RTAS54340.2022.00016

Embedded and real-time systems are increasingly attached to networks. This enables broader coordination beyond the physical system, but also opens the system to attacks. The increasingly complex workloads of these systems include software of varying assurance levels, including that which might be susceptible to compromise by remote attackers. To limit the impact of compromise, μ-kernels focus on maintaining strong memory protection domains between different bodies of software, including system services. They enable limited coordination between processes through Inter-Process Communication (IPC). Real-time systems also require strong temporal guarantees for tasks, and thus need temporal isolation to limit the impact of malicious software. This is challenging as multiple client threads that use IPC to request service from a shared server will impact each other's response times.To constrain the temporal interference between threads, modern μ-kernels often build priority and budget awareness into the system. Unfortunately, this paper demonstrates that this is more challenging than previously thought. Adding priority awareness to IPC processing can lead to significant interference due to the kernel's prioritization logic. Adding budget awareness similarly creates opportunities for interference due to the budget tracking and management operations. In both situations, a Thundering Herd of malicious threads can significantly delay the activation of mission-critical tasks. The Thundering Herd effects are evaluated on seL4 and results demonstrate that high-priority threads can be delayed by over 100,000 cycles per malicious thread. This paper reveals a challenging dilemma: the temporal protections μ-kernels add can, themselves, provide means of threatening temporal isolation. Finally, to defend the system, we identify and empirically evaluate possible mitigations, and propose an admission-control test based upon an interference-aware analysis.