GAP: A novel Generative context-Aware Prompt-tuning method for relation extraction

• Published in: Expert systems with applications Vol. 248; p. 123478
• Main Authors: Chen, Zhenbin, Li, Zhixin, Zeng, Yufei, Zhang, Canlong, Ma, Huifang
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 15.08.2024
• Subjects: Contrastive learning; Few-shot learning; Pretrained language model; Prompt-tuning; Relation extraction; Relation extraction; Pretrained language model; Prompt-tuning; Few-shot learning; Contrastive learning
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2024.123478

Prompt-tuning was proposed to bridge the gap between pretraining and downstream tasks, and it has achieved promising results in Relation Extraction (RE). Although the existing prompt-based RE methods have outperformed the methods based on fine-tuning paradigm, these methods require domain experts to design prompt templates, making them hard to be generalized. In this paper, we propose a Generative context-Aware Prompt-tuning method (GAP) to address these limitations. Our method consists of three crucial modules: (1) a pretrained prompt generator module that extracts or generates the relation triggers from the context and embeds them into the prompt tokens, (2) an in-domain adaptive pretraining module that further trains the Pretrained Language Models (PLMs) to promote the adaptability of the model, and (3) a joint contrastive loss that prevents PLMs from generating unrelated content and optimizes our model more effectively. We observe that the context-enhanced prompt tokens generated by GAP can better guide PLMs to make more accurate predictions. And the in-domain pretraining can effectively inject domain knowledge to enhance the robustness of the model. We conduct experiments on four public RE datasets with supervised and few-shot settings. The experimental results have demonstrated the superiority of GAP over existing benchmark methods and GAP shows remarkable improvements in few-shot settings, with average F1 score enhancements of 3.5%, 2.7%, and 3.4% on the TACRED, TACREV, and Re-TACRED datasets, respectively. Furthermore, GAP still achieved state-of-the-art (SOTA) performance in supervised settings. •We introduced the prompt-tuning framework for relation extraction task.•We proposed a pre-extractor to enhance the stability of the model.•We designed a prompt generator to generate prompt tokens for RE.•We introduced a retrieval-based in-domain pretraining strategy.•We proposed Joint Contrastive Loss to optimize our model.