HedgePeer: A Dataset for Uncertainty Detection in Peer Reviews

• Published in: Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries pp. 1 - 5
• Main Authors: Ghosal, Tirthankar, Varanasi, Kamal Kaushik, Kordoni, Valia
• Format: Conference Proceeding
• Language: English
• Published: ACM 20.06.2022
• Subjects: Hedges; Information retrieval; Libraries; Linguistics; Measurement uncertainty; Peer Reviews; Predictive models; Reviewer Confidence; Uncertainty; Uncertainty Detection; Writing
• DOI: 10.1145/3529372.3533300

Prague, Czech Republic Uncertainty detection from text is essential in many applications in information retrieval (IR). Detecting textual uncertainties helps extract factual information instead of uncertain or non-factual information. To avoid overprecise commitment, people use linguistic devices like hedges (uncertain words or phrases). In peer reviews, reviewers often use hedges wherever they are unsure about their opinion or when facts do not back their opinions. Usage of hedges or uncertain words in writing can also indicate the reviewer's confidence or measure of conviction in their reviews. Reviewer confidence is important in the peer review process (especially to the editors or chairs) to judge the quality of evaluation of the paper under review. However, the self-annotated reviewer confidence score is often miscalibrated or biased and not an accurate representation of the reviewer's conviction of their judgment on the merit of the paper. Less confident reviewers sometimes speculate their observations. Here in this paper, we introduce HedgePeer, a new uncertainty detection dataset of peer review comments, which is more than five times larger than the existing datasets on hedge detection in other domains. We curate our dataset from the open-access reviews available in the open review platform and annotate the review comments in terms of the hedge cues and hedge spans. We also provide several baseline approaches, including a multitask learning model with sentiment intensity and parts-of-speech as scaffold tasks to predict hedge cues and spans. We make our dataset and baseline codes available at https://github.com/Tirthankar-Ghosal/HedgePeer-Dataset. Our dataset is motivated towards computationally estimating the reviewer's conviction from their review texts. CCS CONCEPTS * Computing methodologies → Information extraction; * Information systems → Information extraction.