Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities

• Published in: Sensors (Basel, Switzerland) Vol. 23; no. 22; p. 9287
• Main Authors: Watanabe, Yuto, Togo, Ren, Maeda, Keisuke, Ogawa, Takahiro, Haseyama, Miki
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.11.2023
• Subjects: Accuracy; Computational linguistics; evaluation metric; generative adversarial network; Language processing; manipulation direction; Natural language; Natural language interfaces; Performance evaluation; text-guided image manipulation
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23229287

At present, text-guided image manipulation is a notable subject of study in the vision and language field. Given an image and text as inputs, these methods aim to manipulate the image according to the text, while preserving text-irrelevant regions. Although there has been extensive research to improve the versatility and performance of text-guided image manipulation, research on its performance evaluation is inadequate. This study proposes Manipulation Direction (MD), a logical and robust metric, which evaluates the performance of text-guided image manipulation by focusing on changes between image and text modalities. Specifically, we define MD as the consistency of changes between images and texts occurring before and after manipulation. By using MD to evaluate the performance of text-guided image manipulation, we can comprehensively evaluate how an image has changed before and after the image manipulation and whether this change agrees with the text. Extensive experiments on Multi-Modal-CelebA-HQ and Caltech-UCSD Birds confirmed that there was an impressive correlation between our calculated MD scores and subjective scores for the manipulated images compared to the existing metrics.