Beyond Poisson: Modeling Inter-Arrival Time of Requests in a Datacenter

• Published in: Advances in Knowledge Discovery and Data Mining pp. 198 - 209
• Main Authors: Juan, Da-Cheng, Li, Lei, Peng, Huan-Kai, Marculescu, Diana, Faloutsos, Christos
• Format: Book Chapter
• Language: English
• Published: Cham Springer International Publishing 2014
• Series: Lecture Notes in Computer Science
• Subjects: Cumulative Density Function; Defense Advance Research Project Agency; Discrete Fourier Transform; Little Common Multiple
• ISBN: 9783319066042; 3319066048
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-06605-9_17

How frequently are computer jobs submitted to an industrial-scale datacenter? We investigate the trace that contains job requests and execution collected in one of large-scale industrial datacenters, which spans near half of a Terabyte. In this paper, we discover and explain two surprising patterns with respect to the inter-arrival time (IAT) of job requests: (a) multiple periodicities and (b) multi-level bundling effects. Specifically, we propose a novel generative process, Hierarchical Bundling Model (HiBM), for modeling the data. HiBM is able to mimic multiple components in the distribution of IAT, and to simulate job requests with the same statistical properties as in the real data. We also provide a systematic approach to estimate the parameters of HiBM.