Foundation and large language models: fundamentals, challenges, opportunities, and social impacts

• Published in: Cluster computing Vol. 27; no. 1; pp. 1 - 26
• Main Authors: Myers, Devon, Mohawesh, Rami, Chellaboina, Venkata Ishwarya, Sathvik, Anantha Lakshmi, Venkatesh, Praveen, Ho, Yi-Hui, Henshaw, Hanna, Alhawawreh, Muna, Berdik, David, Jararweh, Yaser
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.02.2024; Springer Nature B.V
• Subjects: Adaptation; Artificial intelligence; Chatbots; Computer Communication Networks; Computer Science; Computers; Industrial development; Language; Large language models; Machine translation; Natural language processing; Neural networks; Operating Systems; Processor Architectures; Advanced pre-trained models; Natural language processing; Foundation models; Large language models; Artificial intelligence; Machine learning
• ISSN: 1386-7857; 1573-7543
• DOI: 10.1007/s10586-023-04203-7

Foundation and Large Language Models (FLLMs) are models that are trained using a massive amount of data with the intent to perform a variety of downstream tasks. FLLMs are very promising drivers for different domains, such as Natural Language Processing (NLP) and other AI-related applications. These models emerged as a result of the AI paradigm shift, involving the use of pre-trained language models (PLMs) and extensive data to train transformer models. FLLMs have also demonstrated impressive proficiency in addressing a wide range of NLP applications, including language generation, summarization, comprehension, complex reasoning, and question answering, among others. In recent years, there has been unprecedented interest in FLLMs-related research, driven by contributions from both academic institutions and industry players. Notably, the development of ChatGPT, a highly capable AI chatbot built around FLLMs concepts, has garnered considerable interest from various segments of society. The technological advancement of large language models (LLMs) has had a significant influence on the broader artificial intelligence (AI) community, potentially transforming the processes involved in the development and use of AI systems. Our study provides a comprehensive survey of existing resources related to the development of FLLMs and addresses current concerns, challenges and social impacts. Moreover, we emphasize on the current research gaps and potential future directions in this emerging and promising field.