Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction

• Published in: BMC bioinformatics Vol. 23; no. 1; pp. 120 - 20
• Main Authors: Su, Peng, Vijay-Shanker, K.
• Format: Journal Article
• Language: English
• Published: London BioMed Central 04.04.2022; BMC
• Subjects: Adaptation; Algorithms; BERT; Bioinformatics; Biomedical and Life Sciences; Biomedical relation extraction; Biomedical Research - methods; Classification; Computational Biology/Bioinformatics; Computer Appl. in Life Sciences; Data Mining - methods; Datasets; Deep learning; Domains; Drug interactions; Electric Power Supplies; Experiments; Knowledge; Language; Life Sciences; Medical research; Microarrays; Natural language; Natural Language Processing; Performance enhancement; Proteins; Semantics; Text mining; Training; Transformer; Deep learning; Text mining; BERT; Transformer; Biomedical relation extraction
• ISSN: 1471-2105; 1471-2105
• DOI: 10.1186/s12859-022-04642-w

Background Recently, automatically extracting biomedical relations has been a significant subject in biomedical research due to the rapid growth of biomedical literature. Since the adaptation to the biomedical domain, the transformer-based BERT models have produced leading results on many biomedical natural language processing tasks. In this work, we will explore the approaches to improve the BERT model for relation extraction tasks in both the pre-training and fine-tuning stages of its applications. In the pre-training stage, we add another level of BERT adaptation on sub-domain data to bridge the gap between domain knowledge and task-specific knowledge. Also, we propose methods to incorporate the ignored knowledge in the last layer of BERT to improve its fine-tuning. Results The experiment results demonstrate that our approaches for pre-training and fine-tuning can improve the BERT model performance. After combining the two proposed techniques, our approach outperforms the original BERT models with averaged F1 score improvement of 2.1% on relation extraction tasks. Moreover, our approach achieves state-of-the-art performance on three relation extraction benchmark datasets. Conclusions The extra pre-training step on sub-domain data can help the BERT model generalization on specific tasks, and our proposed fine-tuning mechanism could utilize the knowledge in the last layer of BERT to boost the model performance. Furthermore, the combination of these two approaches further improves the performance of BERT model on the relation extraction tasks.