Application of Pre-training Models in Named Entity Recognition

Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks...

Full description

Saved in:
Bibliographic Details
Published in2020 12th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) Vol. 1; pp. 23 - 26
Main Authors Wang, Yu, Sun, Yining, Ma, Zuchang, Gao, Lisheng, Xu, Yang, Sun, Ting
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.08.2020
Subjects
BERT
Bit error rate
ERNIE
ERNIE2.0-tiny
Hidden Markov models
Machine learning
named entity recognition
Natural language processing
pre-training model
RoBERTa
Semantics
Task analysis
Training
Online AccessGet full text

Cover

More Information
Summary:Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks. In this paper, firstly, we introduce the architecture and pre-training tasks of four common pre-training models: BERT, ERNIE, ERNIE2.0-tiny, and RoBERTa. Then, we apply these pre-training models to a NER task by fine-tuning, and compare the effects of the different model architecture and pre-training tasks on the NER task. The experiment results showed that RoBERTa achieved state-of-the-art results on the MSRA-2006 dataset.
DOI:10.1109/IHMSC49165.2020.00013