Evaluation on ChatGPT for Chinese Language Understanding

ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension...

Full description

Saved in:
Bibliographic Details
Published inData intelligence Vol. 5; no. 4; pp. 885 - 903
Main Authors Li, Linhan, Zhang, Huaping, Li, Chunjin, You, Haowen, Cui, Wenyao
Format Journal Article
LanguageEnglish
Published Cambridge MIT Press Journals, The 01.11.2023
Subjects
Artificial intelligence
Chatbots
Chinese languages
Data mining
Datasets
Machine translation
Natural language processing
R&D
Reading comprehension
Research & development
Semantics
Sentiment analysis
Online AccessGet full text

Cover

More Information
Summary:ChatGPT has attracted extension attention of academia and industry. This paper aims to evaluate ChatGPT in Chinese language understanding capability on 6 tasks using 11 datasets. Experiments indicate that ChatGPT achieved competitive results in sentiment analysis, summary, and reading comprehension in Chinese, while it is prone to factual errors in closed-book QA. Further, on two more difficult Chinese understanding tasks, that is, idiom fill-in-the-blank and cants understanding, we found that a simple chain-of-thought prompt can improve the accuracy of ChatGPT in complex reasoning. This paper further analyses the possible risks of using ChatGPT based on the results. Finally, we briefly describe the research and development progress of our ChatBIT.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2641-435X
2641-435X
DOI:10.1162/dint_a_00232