Auto-regressive neural network systems with a soft attention mechanism using support data patches

• Main Authors: van den Oord, Aaron Gerard Antonius, Rezende, Danilo Jimenez, Vinyals, Oriol, Gomes de Freitas, Joao Ferdinando, Chen, Yutian, Reed, Scott Ellison
• Format: Patent
• Language: English
• Published: 23.04.2024
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; PHYSICS

A system comprising a causal convolutional neural network to autoregressively generate a succession of values of a data item conditioned upon previously generated values of the data item. The system includes support memory for a set of support data patches each of which comprises an encoding of an example data item. A soft attention mechanism attends to one or more patches when generating the current item value. The soft attention mechanism determines a set of scores for the support data patches, for example in the form of a soft attention query vector dependent upon the previously generated values of the data item. The soft attention query vector is used to query the memory. When generating the value of the data item at a current iteration layers of the causal convolutional neural network are conditioned upon the support data patches weighted by the scores.