ParsBERT: Transformer-based Model for Persian Language Understanding

• Published in: Neural processing letters Vol. 53; no. 6; pp. 3831 - 3847
• Main Authors: Farahani, Mehrdad, Gharachorloo, Mohammad, Farahani, Marzieh, Manthouri, Mohammad
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.12.2021; Springer Nature B.V
• Subjects: Artificial Intelligence; BERT; Complex Systems; Computational Intelligence; Computer Science; Data mining; Datasets; Language; Language Models; Machine learning; Multilingualism; Natural language; Natural language processing; Neural networks; NLP; NLU; Persian; Persian language; Sentiment analysis; Transformers; Persian; Language Models; Transformers; BERT; NLP; NLU
• ISSN: 1370-4621; 1573-773X; 1573-773X
• DOI: 10.1007/s11063-021-10528-4

The surge of pre-trained language models has begun a new era in the field of Natural Language Processing (NLP) by allowing us to build powerful language models. Among these models, Transformer-based models such as BERT have become increasingly popular due to their state-of-the-art performance. However, these models are usually focused on English, leaving other languages to multilingual models with limited resources. This paper proposes a monolingual BERT for the Persian language (ParsBERT), which shows its state-of-the-art performance compared to other architectures and multilingual models. Also, since the amount of data available for NLP tasks in Persian is very restricted, a massive dataset for different NLP tasks as well as pre-training the model is composed. ParsBERT obtains higher scores in all datasets, including existing ones and gathered ones, and improves the state-of-the-art performance by outperforming both multilingual BERT and other prior works in Sentiment Analysis, Text Classification, and Named Entity Recognition tasks.