Deep Multi-Modality Adversarial Networks for Unsupervised Domain Adaptation

• Published in: IEEE transactions on multimedia Vol. 21; no. 9; pp. 2419 - 2431
• Main Authors: Ma, Xinhong, Zhang, Tianzhu, Xu, Changsheng
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.09.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Adaptation; Adaptation models; Correlation; Data mining; Feature extraction; Knowledge management; Labels; multi-modality; Object recognition; Representations; Semantics; social event recognition; stacked attention; Task analysis; Training; triplet loss; Unsupervised domain adaptation
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2019.2902100

Unsupervised domain adaptation aims to transfer domain knowledge from existing well-defined tasks to new ones where labels are unavailable. In the real-world applications, domain discrepancy is usually uncontrollable especially for multi-modality data. Therefore, it is significantly motivated to deal with a multi-modality domain adaptation task. As labels are unavailable in a target domain, how to learn semantic multi-modality representations and successfully adapt the classifier from a source to the target domain remain open challenges in a multi-modality domain adaptation task. To deal with these issues, we propose a multi-modality adversarial network (MMAN), which applies stacked attention to learn semantic multi-modality representations and reduces domain discrepancy via adversarial training. Unlike the previous domain adaptation methods, which cannot make full use of source domain categories information, multi-channel constraint is employed to capture fine-grained categories of knowledge that could enhance the discrimination of target samples and boost target performance on single-modality and multi-modality domain adaptation problems. We apply the proposed MMAN to two applications including cross-domain object recognition and cross-domain social event recognition. The extensive experimental evaluations demonstrate the effectiveness of the proposed model for unsupervised domain adaptation.