A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Information

• Published in: Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online) pp. 13979 - 13989
• Main Authors: Kowal, Matthew, Siam, Mennatullah, Islam, Md Amirul, Bruce, Neil D. B., Wildes, Richard P., Derpanis, Konstantinos G.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01.06.2022
• Subjects: Computational modeling; Computer architecture; Computer vision; Dynamics; grouping and shape analysis; Heuristic algorithms; Object segmentation; Video analysis and understanding; Action and event recognition; Explainable computer vision; Segmentation; Visualization
• ISSN: 1063-6919
• DOI: 10.1109/CVPR52688.2022.01361

Deep spatiotemporal models are used in a variety of computer vision tasks, such as action recognition and video object segmentation. Currently, there is a limited understanding of what information is captured by these models in their intermediate representations. For example, while it has been observed that action recognition algorithms are heavily influenced by visual appearance in single static frames, there is no quantitative methodology for evaluating such static bias in the latent representation compared to bias toward dynamic information (e.g. motion). We tackle this challenge by proposing a novel approach for quantifying the static and dynamic biases of any spatiotemporal model. To show the efficacy of our approach, we analyse two widely studied tasks, action recognition and video object segmentation. Our key findings are threefold: (i) Most examined spatiotemporal models are biased toward static information; although, certain two-stream architectures with cross-connections show a better balance between the static and dynamic information captured. (ii) Some datasets that are commonly assumed to be biased toward dynamics are actually biased toward static information. (iii) Individual units (channels) in an architecture can be biased toward static, dynamic or a combination of the two. 1 1 Project page and code