Rethinking Optimization and Architecture for Tiny Language Models

• Published in: arXiv.org
• Main Authors: Tang, Yehui, Liu, Fangcheng, Ni, Yunsheng, Tian, Yuchuan, Bai, Zheyuan, Yi-Qi, Hu, Liu, Sichao, Shangling Jui, Han, Kai, Wang, Yunhe
• Format: Paper
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 06.02.2024
• Subjects: Design optimization; Design parameters; Empirical analysis; Language; Large language models; Mathematical models; Optimization; Training

The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently...