An Empirical Study on Compliance with Ranking Transparency in the Software Documentation of EU Online Platforms

• Published in: 2024 IEEE/ACM 46th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) pp. 46 - 56
• Main Authors: Sovrano, Francesco, Lognoul, Michael, Bacchelli, Alberto
• Format: Conference Proceeding
• Language: English
• Published: ACM 14.04.2024
• Subjects: Compliance Assessment; Documentation; EU Regulations; Europe; Explainability; Internet; Online platforms; Ranking Transparency; Search engines; Software; Software algorithms; Software Documentation; Sustainable development
• ISSN: 2832-7616

Compliance with the European Union's Platform-to-Business (P2B) Regulation helps fostering a fair, ethical and secure online environment. However, it is challenging for online platforms, and assessing their compliance can be difficult for public authorities. This is partly due to the lack of automated tools for assessing the information (e.g., software documentation) platforms provide concerning ranking transparency. Our study tackles this issue in two ways. First, we empirically evaluate the compliance of six major platforms (Amazon, Bing, Booking, Google, Tripadvisor, and Yahoo), revealing substantial differences in their documentation. Second, we introduce and test automated compliance assessment tools based on ChatGPT and information retrieval technology. These tools are evaluated against human judgments, showing promising results as reliable proxies for compliance assessments. Our findings could help enhance regulatory compliance and align with the United Nations Sustainable Development Goal 10.3, which seeks to reduce inequality, including business disparities, on these platforms.Data and materials: https://doi.org/10.5281/zenodo.10478546.CCS CONCEPTS*Software and its engineering → Documentation;*Information systems →Data analytics;*Appliedcomputing → Law;*Computing methodologies → Natural language processing.LAY ABSTRACTIn the digital world, software powers online platforms such as marketplaces and search engines. For instance, software determines the display order of products on Amazon or hotels on Booking.com. Understanding these systems is vital for businesses, yet the explanations can be complex. The European Union mandates online platforms to reveal the main parameters of their ranking algorithms, aiming for fairness and reducing inequalities. However, ensuring these platforms comply is difficult due to the lack of a standard evaluation method. Our study examined explanations from Google, Yahoo, Bing, Amazon, Booking, and Tripadvisor. We gathered insights from three legal experts and over a hundred people to gauge the clarity of these explanations, finding significant variations in clarity across platforms. Reviewing these explanations manually is cumbersome. To streamline this, we developed automated tools and compared their effectiveness against human assessments. Our discussion revolves around how these tools can enhance transparency on online platforms.