SPDK: A Development Kit to Build High Performance Storage Applications

• Published in: Proceedings (IEEE International Conference on Cloud Computing Technology and Science) pp. 154 - 161
• Main Authors: Ziye Yang, Harris, James R., Walker, Benjamin, Verkamp, Daniel, Changpeng Liu, Cunyin Chang, Gang Cao, Stern, Jonathan, Verma, Vishal, Paul, Luse E.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.12.2017
• Subjects: asynchronous polled mode driven I/O; Hardware; High performance storage application; Inductors; Kernel; Libraries; Nonvolatile memory; Performance evaluation; User space NVMe driver
• ISSN: 2330-2186
• DOI: 10.1109/CloudCom.2017.14

There is strong demand on building high performance storage service upon emerging fast storage devices (e.g., NVMe SSDs). Unfortunately, current storage software stack cannot satisfy such requirements and the software overhead becomes a major bottleneck for developing high performance storage applications. According to our performance profiling results, most storage software overhead is caused by kernel I/O stacks due to context switch, data copy, interrupt, resource synchronization and etc. To address these issues, we provide SPDK (storage performance development kit), a set of tools and libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance by moving the necessary drivers into user space and operating them in a polled mode instead of interrupt mode, which eliminates kernel context switch and interrupt handling overhead and also provides lockless resource access. Integrated with SPDK, performance competitive storage applications can be build upon those fast storage devices. In our experiments, the per cpu core IOPS of NVMe device driver in SPDK is about 6X to 10X better than the kernel NVMe driver. Some storage applications are also developed upon SPDK's framework and obtain much better per-core performance compared with the existing solutions. For example, SPDK NVMe over fabrics (NVMe-oF) target is 10X more efficient than Linux Kernel NVMe-oF target solution.