State Graph Reasoning for Multimodal Conversational Recommendation

• Published in: IEEE transactions on multimedia Vol. 25; pp. 3113 - 3124
• Main Authors: Wu, Yuxia, Liao, Lizi, Zhang, Gangyi, Lei, Wenqiang, Zhao, Guoshuai, Qian, Xueming, Chua, Tat-Seng
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Cognition; Conversation; Data structures; Decision making; Electronic mail; History; knowledge graph; Knowledge representation; Negligence; Reasoning; recommend- ation systems; Recommender systems; Training; User requirements; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2022.3155900

Conversational recommendation system (CRS) attracts increasing attention in various application domains such as retail and travel. It offers an effective way to capture users' dynamic preferences with multi-turn conversations. However, most current studies center on the recommendation aspect while over-simplifying the conversation process. The negligence of complexity in data structure and conversation flow hinders their practicality and utility. In reality, there exist various relationships among slots and values, while users' requirements may dynamically adjust or change. Moreover, the conversation often involves visual modality to facilitate the conversation. These actually call for a more advanced internal state representation of the dialogue and a proper reasoning scheme to guide the decision making process. In this paper, we explore multiple facets of multimodal conversational recommendation and try to address the above mentioned challenges. In particular, we represent the structured back-end database as a multimodal knowledge graph which captures the various relations and evidence in different modalities. The user preferences expressed via conversation utterances will then be gradually updated to the state graph with clear polarity. Based on these, we train an end-to-end S tate G raph-based R easoning model ( SGR ) to perform reasoning over the whole state graph. The prediction of our proposed model benefits from the structure of the graph. It not only allows for zero-shot reasoning for items unseen in training conversations, but also provides a natural way to explain the policies. Extensive experiments show that our model achieves better performance compared with existing methods.