BP-MoE: Behavior Pattern-aware Mixture-of-Experts for Temporal Graph Representation Learning

• Published in: Knowledge-based systems Vol. 299; p. 112056
• Main Authors: Chen, Chonghao, Cai, Fei, Chen, Wanyu, Zheng, Jianming, Zhang, Xin, Luo, Aimin
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 05.09.2024
• Subjects: Graph representation learning; Mixture-of-experts; Temporal graph; Graph representation learning; Temporal graph; Mixture-of-experts
• ISSN: 0950-7051; 1872-7409
• DOI: 10.1016/j.knosys.2024.112056

Temporal graph representation learning aims to develop low-dimensional embeddings for nodes in a graph that can effectively capture their structural and temporal properties. Prior approaches primarily focus on devising sophisticated neural architectures to encode the historical interactions of nodes, enabling capturing the specific behavior patterns. However, such methods often overlook the fact that different nodes in a graph may exhibit distinct evolutionary preferences, resulting in a failure to adequately model the variations in the node evolution behavior. To address these issues, we propose a novel temporal graph learning framework called Behavior Pattern-aware Mixture-of-Experts (BP-MoE), which leverages multiple expert encoder networks with diverse architectures to comprehensively capture the behavior patterns for nodes. In detail, we first design a multi-perspective graph encoder that employs experts based on the gated recurrent units, graph attention units, and multilayer perceptron units to encode a node’s long-term memory, high-order neighborhood features, and short-term behavior preference, respectively. Subsequently, to achieve the personalized evolution preference learning for each individual node, we incorporate a behavior pattern-aware MoE mechanism to activate the pattern-specific experts for each node via a gating network. Finally, to optimize the training process of expert networks, we introduce two auxiliary losses that prevent the imbalanced training across experts. Extensive experiments conducted on multiple benchmark datasets verify the superiority of our approach over the state-of-the-art baselines. Specifically, BP-MoE consistently outperforms the baseline models for the link prediction and node classification tasks under various experimental settings. •Introduce a novel temporal graph representation learning framework to comprehensively model distinct behavior patterns of nodes in a temporal graph.•Propose a behavior pattern-aware mixture-of-expert mechanism to adaptively capture the personalized evolution preferences for nodes.•Verify the method’s superiority in temporal node/edge property prediction tasks across various model settings.