Pre-trained models for natural language processing: A survey

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize...

Full description

Saved in:
Bibliographic Details
Published inScience China. Technological sciences Vol. 63; no. 10; pp. 1872 - 1897
Main Authors Qiu, XiPeng, Sun, TianXiang, Xu, YiGe, Shao, YunFan, Dai, Ning, Huang, XuanJing
Format Journal Article
LanguageEnglish
Published Beijing Science China Press 01.10.2020
Springer Nature B.V
Subjects
Engineering
Natural language processing
Review
Taxonomy
deep learning
language modelling
natural language processing
word embedding
pre-trained model
self-supervised learning
neural network
distributed representation
Online AccessGet full text

Cover

More Information
Summary:Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1674-7321
1869-1900
DOI:10.1007/s11431-020-1647-3