Guided CNN for generalized zero-shot and open-set recognition using visual and semantic prototypes

• Published in: Pattern recognition Vol. 102; p. 107263
• Main Authors: Geng, Chuanxing, Tao, Lue, Chen, Songcan
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.06.2020
• Subjects: Convolutional prototype learning; Generalized zero-shot Learning; Open set recognition; Open set recognition; Convolutional prototype learning; Generalized zero-shot Learning
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2020.107263

•Decomposing G-ZSL task into an OSR task and a ZSL task while jointly.•training the models of these two tasks effectively addresses the CO problem which is ubiquitous in most existing G-ZSL methods.•Introducing the accumulated side information from known classes to OSR to first explore a new generalized open set recognition (G OSR) task.•A visual and semantic prototypes jointly guided convolutional ne ural net work (VSG CNN) is proposed to fulfill these two tasks (G ZSL and G OSR) in a unified end to end learning framework.•Extensive experiments on benchmark datasets indicate the validity of our VSG-CNN. In the process of exploring the world, the curiosity constantly drives humans to cognize new things. Supposing you are a zoologist, for a presented animal image, you can recognize it immediately if you know its class. Otherwise, you would more likely attempt to cognize it by exploiting the side-information (e.g., semantic information, etc.) you have accumulated. Inspired by this, this paper decomposes the generalized zero-shot learning (G-ZSL) task into an open set recognition (OSR) task and a zero-shot learning (ZSL) task, where OSR recognizes seen classes (if we have seen (or known) them) and rejects unseen classes (if we have never seen (or known) them before), while ZSL identifies the unseen classes rejected by the former. Simultaneously, without violating OSR’s assumptions (only known class knowledge is available in training), we also first attempt to explore a new generalized open set recognition (G-OSR) by introducing the accumulated side-information from known classes to OSR. For G-ZSL, such a decomposition effectively solves the class overfitting problem with easily misclassifying unseen classes as seen classes. The problem is ubiquitous in most existing G-ZSL methods. On the other hand, for G-OSR, introducing such semantic information of known classes not only improves the recognition performance but also endows OSR with the cognitive ability of unknown classes. Specifically, a visual and semantic prototypes-jointly guided convolutional neural network (VSG-CNN) is proposed to fulfill these two tasks (G-ZSL and G-OSR) in a unified end-to-end learning framework. Extensive experiments on benchmark datasets demonstrate the advantages of our learning framework.