Network-Wide Traffic Signal Control Based on MARL with Hierarchical Nash-Stackelberg Game Model

• Published in: IEEE access Vol. 11; p. 1
• Main Authors: Shen, Hui, Zhao, Hongxia, Zhang, Zundong, Yang, Xun, Song, Yutong, Liu, Xiaoming
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 01.01.2023; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Approximation algorithms; Game theory; Games; Hierarchical game model; Machine learning; Multi-agent reinforcement learning; Multi-agent systems; Multiagent systems; Network-wide traffic signal control; Optimization; Process control; Q-learning; Reinforcement learning; Roads; Roads & highways; Traffic accidents; Traffic congestion; Traffic control; Traffic flow; Traffic intersections; Traffic models; Traffic signals; Travel time; Urban areas
• DOI: 10.1109/ACCESS.2023.3345448
• ISSN: 2169-3536

Network-wide traffic signal control is an important means of relieving urban congestion, reducing traffic accidents, and improving traffic efficiency. However, solving the problem of computational complexity caused by multi-intersection games is challenging. To address this issue, we propose a Nash-Stackelberg hierarchical game model that considers the importance of different intersections in the road network and the game relationships between intersections. The model takes into account traffic control strategies between and within sub-areas of the road network, with important intersections in the two sub-areas as the game subject at the upper layer and secondary intersections as the game subject at the lower layer. Furthermore, we propose two reinforcement learning algorithms (NSHG-QL and NSHG-DQN) based on the Nash-Stackelberg hierarchical game model to realize coordinated control of traffic signals in urban areas. Experimental results show that, compared to basic game model solving algorithms, NSHG-QL and NSHG-DQN algorithms can reduce the average travel time and time loss of vehicles at intersections, increase average speed and road occupancy, and coordinate secondary intersections to make optimal strategy selections based on satisfying the upper-layer game between important intersections. Moreover, the multi-agent reinforcement learning algorithms based on this hierarchical game model can significantly improve learning performance and convergence.