Exploring Gender Biases in Information Retrieval Relevance Judgement Datasets

• Published in: Advances in Information Retrieval Vol. 12657; pp. 216 - 224
• Main Authors: Bigdeli, Amin, Arabzadeh, Negar, Zihayat, Morteza, Bagheri, Ebrahim
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2021; Springer International Publishing
• Series: Lecture Notes in Computer Science
• DOI: 10.1007/978-3-030-72240-1_18
• ISBN: 9783030722395; 3030722392
• ISSN: 0302-9743

Recent studies in information retrieval have shown that gender biases have found their way into representational and algorithmic aspects of computational models. In this paper, we focus specifically on gender biases in information retrieval gold standard datasets, often referred to as relevance judgements. While not explored in the past, we submit that it is important to understand and measure the extent to which gender biases may be presented in information retrieval relevance judgements primarily because relevance judgements are not only the primary source for evaluating IR techniques but are also widely used for training end-to-end neural ranking methods. As such, the presence of bias in relevance judgements would immediately find its way into how retrieval methods operate in practice. Based on a fine-tuned BERT model, we show how queries can be labelled for gender at scale based on which we label MS MARCO queries. We then show how different psychological characteristics are exhibited within documents associated with gendered queries within the relevance judgement datasets. Our observations show that stereotypical biases are prevalent in relevance judgement documents.