Attribution rollout: a new way to interpret visual transformer

• Published in: Journal of ambient intelligence and humanized computing Vol. 14; no. 1; pp. 163 - 173
• Main Authors: Xu, Li, Yan, Xin, Ding, Weiyue, Liu, Zechao
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.01.2023; Springer Nature B.V
• Subjects: Algorithms; Artificial Intelligence; Back propagation; Computational Intelligence; Computer vision; Decision making; Engineering; Methods; Natural language processing; Original Research; Robotics and Automation; Transformers; User Interfaces and Human Computer Interaction; Visualization; Vision transformer; Integrated gradients; Interpretability; Attention
• ISSN: 1868-5137; 1868-5145
• DOI: 10.1007/s12652-022-04354-2

Transformer-based models are dominating the field of natural language processing and are becoming increasingly popular in the field of computer vision. However, the black box characteristics of transformers seriously hamper their application in certain fields. Prior work relies on the raw attention scores or employs heuristic propagation along with the attention graph. In this work, we propose a new way to visualize model. The method computes attention scores based on attribution and then propagates these attention scores through the layers. This propagation involves attention layers and multi-head attention mechanism. Our method extracts salient dependencies in each layer to visualize prediction results. We benchmark our method on recent visual transformer networks and demonstrate its many advantages over the existing interpretability methods. Our code is available at: https://github.com/yxheartipp/attr-rollout .