Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

• Published in: ACM computing surveys Vol. 55; no. 9; pp. 1 - 35
• Main Authors: Liu, Pengfei, Yuan, Weizhe, Fu, Jinlan, Jiang, Zhengbao, Hayashi, Hiroaki, Neubig, Graham
• Format: Journal Article
• Language: English
• Published: New York, NY ACM 30.09.2023; Association for Computing Machinery
• Subjects: Computer science; Computing methodologies; Natural language processing; Prompt engineering; Strings; Supervised learning; Zero-shot learning; prompting; Pre-trained language models
• ISSN: 0360-0300; 1557-7341
• DOI: 10.1145/3560815

This article surveys and organizes research works in a new paradigm in natural language processing, which we dub “prompt-based learning.” Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x′ that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x̂, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: It allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this article, we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g., the choice of pre-trained language models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts but also release other resources, e.g., a website NLPedia–Pretrain including constantly updated survey and paperlist.