Image Retrieval with Text Manipulation by Local Feature Modification

• Published in: 东华大学学报（英文版） Vol. 40; no. 4; pp. 404 - 409
• Main Authors: ZHA Jianhong, YAN Cairong, ZHANG Yanting, WANG Jun
• Format: Journal Article
• Language: English
• Published: College of Computer Science and Technology,Donghua University,Shanghai 201620,China%College of Fashion and Design,Donghua University,Shanghai 200051,China 31.08.2023
• Subjects: image retrieval; attention; local feature modification; text manipulation
• ISSN: 1672-5220
• DOI: 10.19884/j.1672-5220.202204003

TP183; The demand for image retrieval with text manipulation exists in many fields,such as e-commerce and Internet search.Deep metric learning methods are used by most researchers to calculate the similarity between the query and the candidate image by fusing the global feature of the query image and the text feature.However,the text usually corresponds to the local feature of the query image rather than the global feature.Therefore,in this paper,we propose a framework of image retrieval with text manipulation by local feature modification(LFM-IR)which can focus on the related image regions and attributes and perform modification.A spatial attention module and a channel attention module are designed to realize the semantic mapping between image and text.We achieve excellent performance on three benchmark datasets,namely Color-Shape-Size(CSS),Massachusetts Institute of Technology(MIT)States and Fashion200K(+8.3%,+0.7%and +4.6%in R@1).