Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing

• Main Authors: Zhang, Zhuoran, Li, Yongxiang, Kan, Zijian, Cheng, Keyuan, Hu, Lijie, Wang, Di
• Format: Journal Article
• Language: English
• Published: 08.10.2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Learning
• DOI: 10.48550/arxiv.2410.06331

The locate-then-edit paradigm has shown significant promise for knowledge editing (KE) in Large Language Models (LLMs). While previous methods perform well on single-hop fact recall tasks, they consistently struggle with multi-hop factual recall tasks involving newly edited knowledge. In this paper, leveraging tools in mechanistic interpretability, we first identify that in multi-hop tasks, LLMs tend to retrieve implicit subject knowledge from deeper MLP layers, unlike single-hop tasks, which rely on earlier layers. This distinction explains the poor performance of current methods in multi-hop queries, as they primarily focus on editing shallow layers, leaving deeper layers unchanged. To address this, we propose IFMET, a novel locate-then-edit KE approach designed to edit both shallow and deep MLP layers. IFMET employs multi-hop editing prompts and supplementary sets to locate and modify knowledge across different reasoning stages. Experimental results demonstrate that IFMET significantly improves performance on multi-hop factual recall tasks, effectively overcoming the limitations of previous locate-then-edit methods.