The impact of memory on learning sequence-to-sequence tasks

• Published in: Machine learning: science and technology Vol. 5; no. 1; pp. 15053 - 15068
• Main Authors: Seif, Alireza, Loos, Sarah A M, Tucci, Gennaro, Roldán, Édgar, Goldt, Sebastian
• Format: Journal Article
• Language: English
• Published: Bristol IOP Publishing 01.03.2024
• Subjects: Artificial neural networks; auto-regressive model; Autoregressive models; Learning; memory; Memory tasks; Natural language processing; Neural networks; non-Markovianity; Performance degradation; Performance enhancement; recurrent neural network; sequence-to-sequence task; statistical physics

The recent success of neural networks in natural language processing has drawn renewed attention to learning sequence-to-sequence (seq2seq) tasks. While there exists a rich literature that studies classification and regression tasks using solvable models of neural networks, seq2seq tasks have not ye...