Neuron-Level Knowledge Attribution in Large Language Models

Identifying important neurons for final predictions is essential for understanding the mechanisms of large language models. Due to computational constraints, current attribution techniques struggle to operate at neuron level. In this paper, we propose a static method for pinpointing significant neur...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Yu, Zeping, Ananiadou, Sophia
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 24.09.2024
Subjects
Mathematical models
Parameters
Transformers
France
Online AccessGet full text

Cover

More Information
Summary:Identifying important neurons for final predictions is essential for understanding the mechanisms of large language models. Due to computational constraints, current attribution techniques struggle to operate at neuron level. In this paper, we propose a static method for pinpointing significant neurons. Compared to seven other methods, our approach demonstrates superior performance across three metrics. Additionally, since most static methods typically only identify "value neurons" directly contributing to the final prediction, we propose a method for identifying "query neurons" which activate these "value neurons". Finally, we apply our methods to analyze six types of knowledge across both attention and feed-forward network (FFN) layers. Our method and analysis are helpful for understanding the mechanisms of knowledge storage and set the stage for future research in knowledge editing. The code is available on https://github.com/zepingyu0512/neuron-attribution.
ISSN:2331-8422