Recasting Continual Learning as Sequence Modeling

In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for contin...

Full description

Saved in:
Bibliographic Details
Main Authors Lee, Soochan, Son, Jaehyeon, Kim, Gunhee
Format Journal Article
LanguageEnglish
Published 18.10.2023
Subjects
Computer Science - Artificial Intelligence
Computer Science - Learning
Online AccessGet full text

Cover

More Information
Summary:In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequence modeling problem, allowing advanced sequence models to be utilized for continual learning. Under this formulation, the continual learning process becomes the forward pass of a sequence model. By adopting the meta-continual learning (MCL) framework, we can train the sequence model at the meta-level, on multiple continual learning episodes. As a specific example of our new formulation, we demonstrate the application of Transformers and their efficient variants as MCL methods. Our experiments on seven benchmarks, covering both classification and regression, show that sequence models can be an attractive solution for general MCL.
DOI:10.48550/arxiv.2310.11952