Multi-Agent Decision-Making Modes in Uncertain Interactive Traffic Scenarios via Graph Convolution-Based Deep Reinforcement Learning

• Published in: Sensors (Basel, Switzerland) Vol. 22; no. 12; p. 4586
• Main Authors: Gao, Xin, Li, Xueyuan, Liu, Qi, Li, Zirui, Yang, Fan, Luan, Tian
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 17.06.2022; MDPI
• Subjects: Algorithms; Behavior; connected autonomous vehicles; Decision making; Deep learning; Design; Efficiency; GQN; Graph representations; MDGQN; multi-mode decision-making; Performance evaluation; reward function matrix; Traffic accidents & safety; Traffic models; uncertain highway exit scene; Vehicles
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s22124586

As one of the main elements of reinforcement learning, the design of the reward function is often not given enough attention when reinforcement learning is used in concrete applications, which leads to unsatisfactory performances. In this study, a reward function matrix is proposed for training various decision-making modes with emphasis on decision-making styles and further emphasis on incentives and punishments. Additionally, we model a traffic scene via graph model to better represent the interaction between vehicles, and adopt the graph convolutional network (GCN) to extract the features of the graph structure to help the connected autonomous vehicles perform decision-making directly. Furthermore, we combine GCN with deep Q-learning and multi-step double deep Q-learning to train four decision-making modes, which are named the graph convolutional deep Q-network (GQN) and the multi-step double graph convolutional deep Q-network (MDGQN). In the simulation, the superiority of the reward function matrix is proved by comparing it with the baseline, and evaluation metrics are proposed to verify the performance differences among decision-making modes. Results show that the trained decision-making modes can satisfy various driving requirements, including task completion rate, safety requirements, comfort level, and completion efficiency, by adjusting the weight values in the reward function matrix. Finally, the decision-making modes trained by MDGQN had better performance in an uncertain highway exit scene than those trained by GQN.