Fay: Extensible Distributed Tracing from Kernels to Clusters

• Published in: ACM transactions on computer systems Vol. 30; no. 4; pp. 1 - 35
• Main Authors: ERLINGSSON, Ulfar, PEINADO, Marcus, MIHAI BUDIU, Simon Peter, MAINAR-RUIZ, Gloria
• Format: Journal Article
• Language: English
• Published: New York, NY Association for Computing Machinery 01.11.2012
• Subjects: Agglomeration; Applied sciences; Codes; Coding, codes; Computer science; control theory; systems; Computer systems and distributed systems. User interface; Data processing. List processing. Character string processing; Distributed processing; Exact sciences and technology; Information theory; Information, signal and communications theory; Memory organisation. Data processing; Parallel processing; Programming languages; Signal and communications theory; Software; Software engineering; Studies; Systems stability; Telecommunications and information theory; Performance evaluation; Production system; Measurement; Query; System programming; Instrumentation; Data processing; Languages; Distributed system; Implementation; Optimization; Kernel method; Aggregation; Design; Coding; Parallel processing; Experimentation; Software; Performance; Monitoring; Execution time
• ISSN: 0734-2071; 1557-7333
• DOI: 10.1145/2382553.2382555

Fay is a flexible platform for the efficient collection, processing, and analysis of software execution traces. Fay provides dynamic tracing through use of runtime instrumentation and distributed aggregation within machines and across clusters. At the lowest level, Fay can be safely extended with new tracing primitives, including even untrusted, fully optimized machine code, and Fay can be applied to running user-mode or kernel-mode software without compromising system stability. At the highest level, Fay provides a unified, declarative means of specifying what events to trace, as well as the aggregation, processing, and analysis of those events. We have implemented the Fay tracing platform for Windows and integrated it with two powerful, expressive systems for distributed programming. Our implementation is easy to use, can be applied to unmodified production systems, and provides primitives that allow the overhead of tracing to be greatly reduced, compared to previous dynamic tracing platforms. To show the generality of Fay tracing, we reimplement, in experiments, a range of tracing strategies and several custom mechanisms from existing tracing frameworks. Fay shows that modern techniques for high-level querying and data-parallel processing of disagreggated data streams are well suited to comprehensive monitoring of software execution in distributed systems. Revisiting a lesson from the late 1960s [Deutsch and Grant 1971], Fay also demonstrates the efficiency and extensibility benefits of using safe, statically verified machine code as the basis for low-level execution tracing. Finally, Fay establishes that, by automatically deriving optimized query plans and code for safe extensions, the expressiveness and performance of high-level tracing queries can equal or even surpass that of specialized monitoring tools.