The Impact of Knowledge Distillation on the Energy Consumption and Runtime Efficiency of NLP Models

Context. While models like BERT and GPT are powerful, they require substantial resources. Knowledge distillation can be employed as a technique to enhance their efficiency. Yet, we lack a clear understanding on their performance and energy consumption. This uncertainty is a major concern, especially...

Full description

Saved in:
Bibliographic Details
Published in2024 IEEE/ACM 3rd International Conference on AI Engineering – Software Engineering for AI (CAIN) pp. 129 - 133
Main Authors Yuan, Ye, Zhang, Jingzhi, Zhang, Zongyao, Chen, Kaiwei, Shi, Jiacheng, Stoico, Vincenzo, Malavolta, Ivano
Format Conference Proceeding
LanguageEnglish
Published ACM 14.04.2024
Subjects
Analytical models
Biological system modeling
Computational modeling
Empirical Software Engineering
Energy consumption
Energy-Efficient Software
Green products
Memory management
NLP models
Performance evaluation
Online AccessGet full text
DOI10.1145/3644815.3644966

Cover

More Information
Summary:Context. While models like BERT and GPT are powerful, they require substantial resources. Knowledge distillation can be employed as a technique to enhance their efficiency. Yet, we lack a clear understanding on their performance and energy consumption. This uncertainty is a major concern, especially in practical applications, where these models could strain resources and limit accessibility for developers with limited means. Our drive also comes from the pressing need for environmentally-friendly and sustainable applications in light of growing environmental worries. To address this, it is crucial to accurately measure their energy consumption.Goal. This study aims to determine how Knowledge Distillation affects the energy consumption and performance of NLP models.Method. We benchmark BERT, Distilled-BERT, GPT-2, and Distilled-GPT-2 using three different tasks from 3 different categories selected from a third-party dataset. The energy consumption, CPU utilization, memory utilization, and inference time of the considered NLP models are measured and statistically analyzed.Results. We observed notable differences between the original and the distilled version of the measured NLP models. Distilled versions tend to consume less energy, while distilled GPT-2 uses less CPU. Conclusion. The results of this study highlight the critical impact of model choice on performance and energy consumption metrics. Future research should consider a wider range of distilled models, diverse benchmarks, and deployment environments, as well as explore the ecological footprint of these models, particularly in the context of environmental sustainability.
DOI:10.1145/3644815.3644966