A Vision-Based Attention Deep Q-Network with Prior-Based Knowledge

• Published in: 2023 China Automation Congress (CAC) pp. 6155 - 6160
• Main Authors: Ma, Jialin, Li, Ce, Hong, Liang, Wei, Kailun, Zhao, Shutian, Jiang, Hangfei
• Format: Conference Proceeding
• Language: English
• Published: IEEE 17.11.2023
• Subjects: atari games; Decision making; deep reinforcement learning; Games; Optimization; prior-based; self-attention; Solid modeling; Solids; Task analysis; Training; Vision-based attention
• DOI: 10.1109/CAC59555.2023.10451132

In order to unveil the intrinsic workings of deep reinforcement learning(DRL) models and explain the regions of interest attended by the agent during the decision-making process, vision-based RL employs attention mechanisms. However, due to policy optimization leading to changes in the data domain, the agent may even fail to learn a policy. To address this, a vision-based attention deep Q-network(VADQN) method with a prior-based mechanism is proposed. Firstly, prior attention maps are obtained using a learnable Gaussian filter and spectral residual method. Nextly, the attention maps are fine-tuned using a self-attention mechanism to improve their performance. During RL training, both the attention maps and the parameters of the policy network are simultaneously trained to ensure explanations of the regions of interest during online training. Finally, a series of ablation experiments were conducted on atari games to compare the proposed method with human, nature convolutional neural network, and other approaches. The results demonstrate that our proposed method not only reveals the regions of interest attended by DRL during the decision-making process but also enhances DRL performance in certain scenarios.