Multi-label Learning with Missing Labels Using Mixed Dependency Graphs

• Published in: International journal of computer vision Vol. 126; no. 8; pp. 875 - 896
• Main Authors: Wu, Baoyuan, Jia, Fan, Liu, Wei, Ghanem, Bernard, Lyu, Siwei
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.08.2018; Springer; Springer Nature B.V
• Subjects: Artificial Intelligence; Coding; Computer Imaging; Computer Science; Decomposition; Formulations; Graph theory; Image annotation; Image management; Image Processing and Computer Vision; Image retrieval; Labels; Pattern Recognition; Pattern Recognition and Graphics; Semantics; Similarity; Smoothness; Vision; Multi-label learning; Mixed dependency graphs; Image retrieval; Missing labels; Image annotation
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-018-1085-3

This work focuses on the problem of multi-label learning with missing labels (MLML), which aims to label each test instance with multiple class labels given training instances that have an incomplete/partial set of these labels (i.e., some of their labels are missing). The key point to handle missing labels is propagating the label information from the provided labels to missing labels, through a dependency graph that each label of each instance is treated as a node. We build this graph by utilizing different types of label dependencies. Specifically, the instance-level similarity is served as undirected edges to connect the label nodes across different instances and the semantic label hierarchy is used as directed edges to connect different classes. This base graph is referred to as the mixed dependency graph, as it includes both undirected and directed edges. Furthermore, we present another two types of label dependencies to connect the label nodes across different classes. One is the class co-occurrence, which is also encoded as undirected edges. Combining with the above base graph, we obtain a new mixed graph, called mixed graph with co-occurrence (MG-CO). The other is the sparse and low rank decomposition of the whole label matrix, to embed high-order dependencies over all labels. Combining with the base graph, the new mixed graph is called as MG-SL (mixed graph with sparse and low rank decomposition). Based on MG-CO and MG-SL, we further propose two convex transductive formulations of the MLML problem, denoted as MLMG-CO and MLMG-SL respectively. In both formulations, the instance-level similarity is embedded through a quadratic smoothness term, while the semantic label hierarchy is used as a linear constraint. In MLMG-CO, the class co-occurrence is also formulated as a quadratic smoothness term, while the sparse and low rank decomposition is incorporated into MLMG-SL, through two additional matrices (one is assumed as sparse, and the other is assumed as low rank) and an equivalence constraint between the summation of this two matrices and the original label matrix. Interestingly, two important applications, including image annotation and tag based image retrieval, can be jointly handled using our proposed methods. Experimental results on several benchmark datasets show that our methods lead to significant improvements in performance and robustness to missing labels over the state-of-the-art methods.