An adaptive approach to noisy annotations in scientific information extraction

Despite recent advances in large language models (LLMs), the best effectiveness in information extraction (IE) is still achieved by fine-tuned models, hence the need for manually annotated datasets to train them. However, collecting human annotations for IE, especially for scientific IE, where exper...

Full description

Saved in:
Bibliographic Details
Published inInformation processing & management Vol. 61; no. 6; p. 103857
Main Authors Bölücü, Necva, Rybinski, Maciej, Dai, Xiang, Wan, Stephen
Format Journal Article
LanguageEnglish
Published Elsevier Ltd 01.11.2024
Subjects
Dataset
Information extraction
Mislabelled
Noisy
Scientific
Weighted weakly supervised learning
Weighted weakly supervised learning
Mislabelled
Noisy
Scientific
Information extraction
Dataset
Online AccessGet full text

Cover

More Information
Summary:Despite recent advances in large language models (LLMs), the best effectiveness in information extraction (IE) is still achieved by fine-tuned models, hence the need for manually annotated datasets to train them. However, collecting human annotations for IE, especially for scientific IE, where expert annotators are often required, is expensive and time-consuming. Another issue widely discussed in the IE community is noisy annotations. Mislabelled training samples can hamper the effectiveness of trained models. In this paper, we propose a solution to alleviate problems originating from the high cost and difficulty of the annotation process. Our method distinguishes clean training samples from noisy samples and then employs weighted weakly supervised learning (WWSL) to leverage noisy annotations. Evaluation of Named Entity Recognition (NER) and Relation Classification (RC) tasks in Scientific IE demonstrates the substantial impact of detecting clean samples. Experimental results highlight that our method, utilising clean and noisy samples with WWSL, outperforms the baseline RoBERTa on NER (＋4.28, ＋4.59, ＋29.27, and ＋5.21 gain for the ADE, SciERC, STEM-ECR, and WLPC datasets, respectively) and the RC (＋6.09 and ＋4.39 gain for the SciERC and WLPC datasets, respectively) tasks. Comprehensive analyses of our method reveal its advantages over state-of-the-art denoising baseline models in scientific NER. Moreover, the framework is general enough to be adapted to different NLP tasks or domains, which means it could be useful in the broader NLP community.
ISSN:0306-4573
DOI:10.1016/j.ipm.2024.103857