Regularized Optimal Transport Layers for Generalized Global Pooling Operations

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 45; no. 12; pp. 15426 - 15444
• Main Authors: Xu, Hongteng, Cheng, Minjie
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.12.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Bregman ADMM; Data integration; Feature extraction; Global pooling; graph embedding; Graphical representations; Image classification; Machine learning; Neural networks; Optimization; regularized optimal transport; set representation; Signal processing algorithms; Sinkhorn scaling; Solids; Structured data; Task analysis
• ISSN: 0162-8828; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2023.3314661

Global pooling is one of the most significant operations in many machine learning models and tasks, which works for information fusion and structured data (like sets and graphs) representation. However, without solid mathematical fundamentals, its practical implementations often depend on empirical mechanisms and thus lead to sub-optimal, even unsatisfactory performance. In this work, we develop a novel and generalized global pooling framework through the lens of optimal transport. The proposed framework is interpretable from the perspective of expectation-maximization. Essentially, it aims at learning an optimal transport across sample indices and feature dimensions, making the corresponding pooling operation maximize the conditional expectation of input data. We demonstrate that most existing pooling methods are equivalent to solving a regularized optimal transport (ROT) problem with different specializations, and more sophisticated pooling operations can be implemented by hierarchically solving multiple ROT problems. Making the parameters of the ROT problem learnable, we develop a family of regularized optimal transport pooling (ROTP) layers. We implement the ROTP layers as a new kind of deep implicit layer. Their model architectures correspond to different optimization algorithms. We test our ROTP layers in several representative set-level machine learning scenarios, including multi-instance learning (MIL), graph classification, graph set representation, and image classification. Experimental results show that applying our ROTP layers can reduce the difficulty of the design and selection of global pooling - our ROTP layers may either imitate some existing global pooling methods or lead to some new pooling layers fitting data better.