Information-Theoretic Semi-Supervised Metric Learning via Entropy Regularization

• Published in: Neural computation Vol. 26; no. 8; pp. 1717 - 1762
• Main Authors: Niu, Gang, Dai, Bo, Yamada, Makoto, Sugiyama, Masashi
• Format: Journal Article
• Language: English
• Published: One Rogers Street, Cambridge, MA 02142-1209, USA MIT Press 01.08.2014; MIT Press Journals, The
• Subjects: Algorithms; Artificial Intelligence; Brain; Comparative analysis; Computation; Entropy; Entropy (Information theory); Information Theory; Learning; Letters; Manifolds; Optimization; Optimization algorithms; Projection; Regularization
• ISSN: 0899-7667; 1530-888X
• DOI: 10.1162/NECO_a_00614

We propose a general information-theoretic approach to semi-supervised metric learning called (SEmi-supervised metRic leArning Paradigm with Hypersparsity) that does not rely on the manifold assumption. Given the probability parameterized by a Mahalanobis distance, we maximize its entropy on labeled data and minimize its entropy on unlabeled data following entropy regularization. For metric learning, entropy regularization improves manifold regularization by considering the dissimilarity information of unlabeled data in the unsupervised part, and hence it allows the supervised and unsupervised parts to be integrated in a natural and meaningful way. Moreover, we regularize by trace-norm regularization to encourage low-dimensional projections associated with the distance metric. The nonconvex optimization problem of SERAPH could be solved efficiently and stably by either a gradient projection algorithm or an EM-like iterative algorithm whose M-step is convex. Experiments demonstrate that compares favorably with many well-known metric learning methods, and the learned Mahalanobis distance possesses high discriminability even under noisy environments.