True Few-Shot Learning with Prompts—A Real-World Perspective

• Published in: Transactions of the Association for Computational Linguistics Vol. 10; pp. 716 - 731
• Main Authors: Schick, Timo, Schütze, Hinrich
• Format: Journal Article
• Language: English
• Published: One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA MIT Press 17.06.2022; MIT Press Journals, The; The MIT Press
• Subjects: Classification; Human performance; Learning; Linguistics; Machine learning; Natural language processing
• ISSN: 2307-387X; 2307-387X
• DOI: 10.1162/tacl_a_00485

Prompt-based approaches excel at few-shot learning. However, Perez et al. ( ) recently cast doubt on their performance as they had difficulty getting good results in a “true” few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of , a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, performs strongly in true few-shot settings without a dev set. Crucial for this strong performance is a number of design choices, including ’s ability to intelligently handle multiple prompts. We put our findings to a real-world test by running on RAFT, a benchmark of tasks taken from realistic NLP applications for which no labeled dev or test sets are available. achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners can successfully be applied in true few-shot settings and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.