How Can We Know What Language Models Know?

• Published in: Transactions of the Association for Computational Linguistics Vol. 8; pp. 423 - 438
• Main Authors: Jiang, Zhengbao, Xu, Frank F., Araki, Jun, Neubig, Graham
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.01.2020; MIT Press Journals, The; The MIT Press
• Subjects: Accuracy; Archives & records; Computational linguistics; Knowledge; Language; Language modeling; Linguistics; Lower bounds; Natural language; Probability; Profession; Queries

Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ ”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ __ ” may result in more accurately...