How Robust is Unsupervised Representation Learning to Distribution Shift?
The robustness of machine learning algorithms to distributions shift is primarily discussed in the context of supervised learning (SL). As such, there is a lack of insight on the robustness of the representations learned from unsupervised methods, such as self-supervised learning (SSL) and auto-enco...
Saved in:
Main Authors | , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
17.06.2022
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | The robustness of machine learning algorithms to distributions shift is
primarily discussed in the context of supervised learning (SL). As such, there
is a lack of insight on the robustness of the representations learned from
unsupervised methods, such as self-supervised learning (SSL) and auto-encoder
based algorithms (AE), to distribution shift. We posit that the input-driven
objectives of unsupervised algorithms lead to representations that are more
robust to distribution shift than the target-driven objective of SL. We verify
this by extensively evaluating the performance of SSL and AE on both synthetic
and realistic distribution shift datasets. Following observations that the
linear layer used for classification itself can be susceptible to spurious
correlations, we evaluate the representations using a linear head trained on a
small amount of out-of-distribution (OOD) data, to isolate the robustness of
the learned representations from that of the linear head. We also develop
"controllable" versions of existing realistic domain generalisation datasets
with adjustable degrees of distribution shifts. This allows us to study the
robustness of different learning algorithms under versatile yet realistic
distribution shift conditions. Our experiments show that representations
learned from unsupervised learning algorithms generalise better than SL under a
wide variety of extreme as well as realistic distribution shifts. |
---|---|
DOI: | 10.48550/arxiv.2206.08871 |