A Collection of Software Engineering Challenges for Big Data System Development

• Published in: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) pp. 362 - 369
• Main Authors: Hummel, Oliver, Eichelberger, Holger, Giloj, Andreas, Werle, Dominik, Schmid, Klaus
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.08.2018
• Subjects: Big Data; Challenges; Computer architecture; Data analysis; Data privacy; Software; Software Development; Software engineering; Systematics
• DOI: 10.1109/SEAA.2018.00066

In recent years, the development of systems for processing and analyzing large amounts of data (so-called Big Data) has become an important sub-discipline of software engineering. However, to date there exits no comprehensive summary of the specific idiosyncrasies and challenges that the development of Big Data systems imposes on software engineers. With this paper, we aim to provide a first step towards filling this gap based on our collective experience from industry and academic projects as well as from consulting and initial literature reviews. The main contribution of our work is a concise summary of 26 challenges in engineering Big Data systems, collected and consolidated by means of a systematic identification process. The aim is to make practitioners more aware of common challenges and to offer researchers a solid baseline for identifying novel software engineering research directions.