Making data platforms smarter with MOSES

• Published in: Future generation computer systems Vol. 125; pp. 299 - 313
• Main Authors: Francia, Matteo, Gallinucci, Enrico, Golfarelli, Matteo, Leoni, Anna Giulia, Rizzi, Stefano, Santolini, Nicola
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.12.2021
• Subjects: Big data; Data lake; Data platform; Metadata; Data lake; Metadata; Big data; Data platform
• ISSN: 0167-739X; 1872-7115
• DOI: 10.1016/j.future.2021.06.031

The rise of data platforms has enabled the collection and processing of huge volumes of data, but has opened to the risk of losing their control. Collecting proper metadata about raw data and transformations can significantly reduce this risk. In this paper we propose MOSES, a technology-agnostic, extensible, and customizable framework for metadata handling in big data platforms. The framework hinges on a metadata repository that stores information about the objects in the big data platform and the processes that transform them. MOSES provides a wide range of functionalities to different types of users of the platform. Differently from previous high-level proposals, MOSES is fully implemented and it was not conceived for a specific technology. Besides discussing the rationale and the features of MOSES, in this paper we describe its implementation and we test it on a real case study. The ultimate goal is to take a significant step forward towards proving that metadata handling in big data platforms is feasible and beneficial. •MOSES is a framework for metadata handling in big data platforms. MOSES provides a wide range of functionalities to different types of users of the platform.•MOSES is fully implemented and it has been tested on real case studies.•The goal is to prove that metadata handling in big data platforms is feasible and beneficial.