In-Context Retrieval-Augmented Language Models

• Published in: Transactions of the Association for Computational Linguistics Vol. 11; pp. 1316 - 1331
• Main Authors: Ram, Ori, Levine, Yoav, Dalmedigos, Itay, Muhlgay, Dor, Shashua, Amnon, Leyton-Brown, Kevin, Shoham, Yoav
• Format: Journal Article
• Language: English
• Published: Cambridge MIT Press Journals, The 13.11.2023; The MIT Press
• Subjects: Automatic text generation; Context; Documents; Language; Language modeling; Linguistics; Modelling; Reading; Retrieval

Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text gene...