H2O-Danube3 Technical Report

• Main Authors: Pfeiffer, Pascal, Singer, Philipp, Babakhin, Yauhen, Fodor, Gabor, Dhankhar, Nischay, Ambati, Sri Satish
• Format: Journal Article
• Language: English
• Published: 12.07.2024
• Subjects: Computer Science - Computation and Language; Computer Science - Learning

We present H2O-Danube3, a series of small language models consisting of H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T tokens. Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final s...