Attribution rollout: a new way to interpret visual transformer - Xu, Li; Yan, Xin; Ding, Weiyue; Liu, Zechao (Journal of ambient intelligence and humanized computing, 2023)
Transformer-based models are dominating the field of natural language processing and are becoming increasingly popular in the field of computer vision. However, the black box characteristics of transformers seriously hamper their application in certain fields. Prior work relies on the raw attention scores or employs heuristic propagation along with the attention graph. In this work, we propose a new way to visualize model. The method computes attention scores based on attribution and then propagates these attention scores through the layers. This propagation involves attention layers and multi-head attention mechanism. Our method extracts salient dependencies in each layer to visualize prediction results. We benchmark our method on recent visual transformer networks and demonstrate its many advantages over the existing interpretability methods. Our code is available at: https://github.com/yxheartipp/attr-rollout .