Video understanding : Tagging of videos through self attentive learnable key descriptors

• Published in: 2023 11th International Symposium on Electronic Systems Devices and Computing (ESDC) Vol. 1; pp. 1 - 7
• Main Authors: Darapaneni, Narayana, Paduri, Anwesh Reddy, Thomas, Dinu, U, Jisha C, Shrivastava, Abhinao, Biradar, Seema
• Format: Conference Proceeding
• Language: English
• Published: IEEE 04.05.2023
• Subjects: Computational modeling; Self-Attentive NetVLAD; Surveys; Tagging; Training; User-generated content; Video on demand; Video Tagging; Visualization; YouTube-8M
• DOI: 10.1109/ESDC56251.2023.10149869

In today's world, the UGC (User Generated Contents) videos have increased exponentially. Billions of videos are uploaded, played and exchanged between different actors. In this context, automatic video content classification has become a critical and challenging problem, especially in areas like video-based search, recommendation etc. In this work we try to extract frame-level visual and audio features, pre-extracted features are then converted into a compact video level representation effectively and efficiently. We aim to classify the video into a set of categories with high accuracy. From the literature survey, we identified that, the tagging of videos has been a problem which has not reached its maturity yet, and there are many researches happening in this area. It is observed that, the clustering based video description methodologies show a better result compared to the temporal algorithms. We also have identified that, majority of the SOTA techniques use the VLAD (Vector of Locally Aggregated Descriptors) technique to extract the video features and make the codebook learnable through some adjustments introduced in the NetVLAD. The key descriptors would be mostly noisy, and many of them are insignificant. In this work we aim to cascade a Self-Attention Block on the NetVLAD which can extract the significant descriptors and filter out the Noise. The YouTube 8M dataset shall be used for training the model and performance will be compared with other SOTA techniques. Like other similar works, model performance will be measured by GAP Metric (Global Average Precision) for all the videos predicted labels. We aim to achieve a GAP score close to 85% for this work.