DQNViz: A Visual Analytics Approach to Understand Deep Q-Networks

• Published in: IEEE transactions on visualization and computer graphics Vol. 25; no. 1; pp. 288 - 298
• Main Authors: Wang, Junpeng, Gou, Liang, Shen, Han-Wei, Yang, Hao
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.01.2019; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Analytical models; Analytics; Computer Science; Data visualization; Deep Q-Network (DQN); Games; Intelligent agents; Learning (artificial intelligence); Machine learning; Mathematical analysis; model interpretation; reinforcement learning; Training; Visual analytics
• ISSN: 1077-2626; 1941-0506; 1941-0506
• DOI: 10.1109/TVCG.2018.2864504

Deep Q-Network (DQN), as one type of deep reinforcement learning model, targets to train an intelligent agent that acquires optimal actions while interacting with an environment. The model is well known for its ability to surpass professional human players across many Atari 2600 games. Despite the superhuman performance, in-depth understanding of the model and interpreting the sophisticated behaviors of the DQN agent remain to be challenging tasks, due to the long-time model training process and the large number of experiences dynamically generated by the agent. In this work, we propose DQNViz, a visual analytics system to expose details of the blind training process in four levels, and enable users to dive into the large experience space of the agent for comprehensive analysis. As an initial attempt in visualizing DQN models, our work focuses more on Atari games with a simple action space, most notably the Breakout game. From our visual analytics of the agent's experiences, we extract useful action/reward patterns that help to interpret the model and control the training. Through multiple case studies conducted together with deep learning experts, we demonstrate that DQNViz can effectively help domain experts to understand, diagnose, and potentially improve DQN models.