Understanding the Limits of Passive Realtime Datacenter Fault Detection and Localization

• Published in: IEEE/ACM transactions on networking Vol. 27; no. 5; pp. 2001 - 2014
• Main Authors: Roy, Arjun, Das, Rajdeep, Zeng, Hongyi, Bagga, Jasmeet, Snoeren, Alex C.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.10.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Circuit faults; Computer network reliability; Data analysis; Data centers; Facebook; Failure rates; Fault detection; Fault diagnosis; Hardware; Localization; Monitoring; Outliers (statistics); Production; Statistical analysis; Switches
• ISSN: 1063-6692; 1558-2566
• DOI: 10.1109/TNET.2019.2938228

Datacenters are characterized by large scale, stringent reliability requirements, and significant application diversity. However, the realities of employing hardware with non-zero failure rates mean that datacenters are subject to significant numbers of failures that can impact performance. Moreover, failures are not always obvious; network components can fail partially, dropping or delaying only subsets of packets. Thus, traditional fault detection techniques involving end-host or router-based statistics can fall short in their ability to identify these errors. We describe how to expedite the process of detecting and localizing partial datacenter faults using an end-host method generalizable to most datacenter applications. In particular, we correlate end-host transport-layer flow metrics with per-flow network paths and apply statistical analysis techniques to identify outliers and localize faulty links and/or switches. We evaluate our approach in a production Facebook front-end datacenter, focusing on its effectiveness across a range of traffic patterns.