Processing images using self-attention based neural networks

• Main Authors: HOULSBY, Neil Matthew Tinmouth, BEYER, Lucas Klaus, HEIGOLD, Georg, DEGHANI, Mostafa, GELLY, Sylvain, WEISSENBORN, Dirk, KOLESNIKOV, Alexander, MINDERER, Matthias Johannes Lorenz, DOSOVITSKIY, Alexey, UNTERTHINER, Thomas, USZKOREIT, Jakob D, ZHAI, Xiaohua
• Format: Patent
• Language: English
• Published: 24.11.2022
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; PHYSICS

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using self-attention based neural networks. One of the methods includes obtaining one or more images comprising a plurality of pixels; determining, for each image of the one or more images, a plurality of image patches of the image, wherein each image patch comprises a different subset of the pixels of the image; processing, for each image of the one or more images, the corresponding plurality of image patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, wherein a plurality of the input elements correspond to respective different image patches; and processing the input sequences using a neural network to generate a network output that characterizes the one or more images, wherein the neural network comprises one or more self-attention neural network layers.