Self-Labeling Refinement for Robust Representation Learning with Bootstrap Your Own Latent

In this work, we have worked towards two major goals. Firstly, we have investigated the importance of Batch Normalisation (BN) layers in a non-contrastive representation learning framework called Bootstrap Your Own Latent (BYOL). We conducted several experiments to conclude that BN layers are not ne...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Garg, Siddhant, Jain, Dhruval
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 09.04.2022
Subjects
Learning
Representations
Similarity
Trigonometric functions
Online AccessGet full text

Cover

More Information
Summary:In this work, we have worked towards two major goals. Firstly, we have investigated the importance of Batch Normalisation (BN) layers in a non-contrastive representation learning framework called Bootstrap Your Own Latent (BYOL). We conducted several experiments to conclude that BN layers are not necessary for representation learning in BYOL. Moreover, BYOL only learns from the positive pairs of images but ignores other semantically similar images in the same input batch. For the second goal, we have introduced two new loss functions to determine the semantically similar pairs in the same input batch of images and reduce the distance between their representations. These loss functions are Cross-Cosine Similarity Loss (CCSL) and Cross-Sigmoid Similarity Loss (CSSL). Using the proposed loss functions, we are able to surpass the performance of Vanilla BYOL (71.04%) by training the BYOL framework using CCSL loss (76.87%) on the STL10 dataset. BYOL trained using CSSL loss performs comparably with Vanilla BYOL.
ISSN:2331-8422