Model-Protected Multi-Task Learning

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 44; no. 2; pp. 1002 - 1019
• Main Authors: Liang, Jian, Liu, Ziqi, Zhou, Jiayu, Jiang, Xiaoqian, Zhang, Changshui, Wang, Fei
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.02.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Covariance matrices; Covariance matrix; Data models; differential privacy; Learning; low-rank subspace learning; Machine learning; model protection; Multi-task learning; Perturbation; Privacy; Resource management; Security; Task analysis
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2020.3015859

Multi-task learning (MTL) refers to the paradigm of learning multiple related tasks together. In contrast, in single-task learning (STL) each individual task is learned independently. MTL often leads to better trained models because they can leverage the commonalities among related tasks. However, because MTL algorithms can "leak" information from different models across different tasks, MTL poses a potential security risk. Specifically, an adversary may participate in the MTL process through one task and thereby acquire the model information for another task. The previously proposed privacy-preserving MTL methods protect data instances rather than models, and some of them may underperform in comparison with STL methods. In this paper, we propose a privacy-preserving MTL framework to prevent information from each model leaking to other models based on a perturbation of the covariance matrix of the model matrix. We study two popular MTL approaches for instantiation, namely, learning the low-rank and group-sparse patterns of the model matrix. Our algorithms can be guaranteed not to underperform compared with STL methods. We build our methods based upon tools for differential privacy, and privacy guarantees, utility bounds are provided, and heterogeneous privacy budgets are considered. The experiments demonstrate that our algorithms outperform the baseline methods constructed by existing privacy-preserving MTL methods on the proposed model-protection problem.