Can Mamba Always Enjoy the "Free Lunch"?

• Main Authors: Ren, Ruifeng, Li, Zhicong, Liu, Yong
• Format: Journal Article
• Language: English
• Published: 04.10.2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Learning

Transformers have been the cornerstone of current Large Language Models (LLMs); however, its linear growth in overhead during inference with respect to sequence length poses challenges for modeling long sequences. In this context, Mamba has gradually attracted attention due to its constant-level siz...