LieTransformer: Equivariant self-attention for Lie Groups

Group equivariant neural networks are used as building blocks of group invariant neural networks, which have been shown to improve generalisation performance and data efficiency through principled parameter sharing. Such works have mostly focused on group equivariant convolutions, building on the re...

Full description

Saved in:
Bibliographic Details
Published inarXiv.org
Main Authors Hutchinson, Michael, Charline Le Lan, Zaidi, Sheheryar, Dupont, Emilien, Teh, Yee Whye, Kim, Hyunjik
Format Paper
LanguageEnglish
Published Ithaca Cornell University Library, arXiv.org 16.06.2021
Subjects
Lie groups
Neural networks
Particle trajectories
Subgroups
Three dimensional models
Online AccessGet full text

Cover

More Information
Summary:Group equivariant neural networks are used as building blocks of group invariant neural networks, which have been shown to improve generalisation performance and data efficiency through principled parameter sharing. Such works have mostly focused on group equivariant convolutions, building on the result that group equivariant linear maps are necessarily convolutions. In this work, we extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups. We demonstrate the generality of our approach by showing experimental results that are competitive to baseline methods on a wide range of tasks: shape counting on point clouds, molecular property regression and modelling particle trajectories under Hamiltonian dynamics.
ISSN:2331-8422