The Impact of Knowledge Distillation on the Energy Consumption and Runtime Efficiency of NLP Models

• Published in: 2024 IEEE/ACM 3rd International Conference on AI Engineering – Software Engineering for AI (CAIN) pp. 129 - 133
• Main Authors: Yuan, Ye, Zhang, Jingzhi, Zhang, Zongyao, Chen, Kaiwei, Shi, Jiacheng, Stoico, Vincenzo, Malavolta, Ivano
• Format: Conference Proceeding
• Language: English
• Published: ACM 14.04.2024
• Subjects: Analytical models; Biological system modeling; Computational modeling; Empirical Software Engineering; Energy consumption; Energy-Efficient Software; Green products; Memory management; NLP models; Performance evaluation
• DOI: 10.1145/3644815.3644966

Context. While models like BERT and GPT are powerful, they require substantial resources. Knowledge distillation can be employed as a technique to enhance their efficiency. Yet, we lack a clear understanding on their performance and energy consumption. This uncertainty is a major concern, especially in practical applications, where these models could strain resources and limit accessibility for developers with limited means. Our drive also comes from the pressing need for environmentally-friendly and sustainable applications in light of growing environmental worries. To address this, it is crucial to accurately measure their energy consumption.Goal. This study aims to determine how Knowledge Distillation affects the energy consumption and performance of NLP models.Method. We benchmark BERT, Distilled-BERT, GPT-2, and Distilled-GPT-2 using three different tasks from 3 different categories selected from a third-party dataset. The energy consumption, CPU utilization, memory utilization, and inference time of the considered NLP models are measured and statistically analyzed.Results. We observed notable differences between the original and the distilled version of the measured NLP models. Distilled versions tend to consume less energy, while distilled GPT-2 uses less CPU. Conclusion. The results of this study highlight the critical impact of model choice on performance and energy consumption metrics. Future research should consider a wider range of distilled models, diverse benchmarks, and deployment environments, as well as explore the ecological footprint of these models, particularly in the context of environmental sustainability.