World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data

• Published in: Empirical software engineering : an international journal Vol. 26; no. 2
• Main Authors: Ma, Yuxing, Dey, Tapajit, Bogart, Chris, Amreen, Sadika, Valiev, Marat, Tutko, Adam, Kennard, David, Zaretzki, Russell, Mockus, Audris
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.03.2021; Springer Nature B.V
• Subjects: Compilers; Computer Science; Control data (computers); Evaluation; Interpreters; Mining Software Repositories (MSR); Open source software; Programming Languages; Reliability engineering; Social networks; Software Engineering/Programming and Operating Systems; Version control; Workflow; Software mining; Software supply chain; Software ecosystem
• ISSN: 1382-3256; 1573-7616
• DOI: 10.1007/s10664-020-09905-9

Open source software (OSS) is essential for modern society and, while substantial research has been done on individual (typically central) projects, only a limited understanding of the periphery of the entire OSS ecosystem exists. For example, how are the tens of millions of projects in the periphery interconnected through technical dependencies, code sharing, or knowledge flow? To answer such questions we: a) create a very large and frequently updated collection of version control data in the entire FLOSS ecosystems named World of Code (WoC), that can completely cross-reference authors, projects, commits, blobs, dependencies, and history of the FLOSS ecosystems and b) provide capabilities to efficiently correct, augment, query, and analyze that data. Our current WoC implementation is capable of being updated on a monthly basis and contains over 18B Git objects. To evaluate its research potential and to create vignettes for its usage, we employ WoC in conducting several research tasks. In particular, we find that it is capable of supporting trend evaluation, ecosystem measurement, and the determination of package usage. We expect WoC to spur investigation into global properties of OSS development leading to increased resiliency of the entire OSS ecosystem. Our infrastructure facilitates the discovery of key technical dependencies, code flow, and social networks that provide the basis to determine the structure and evolution of the relationships that drive FLOSS activities and innovation.