Efficient Content-Based Sparse Attention with Routing Transformers - Roy, Aurko; Saffar, Mohammad; Vaswani, Ashish; Grangier, David (Transactions of the Association for Computational Linguistics, 2021)
Keywords: Attention; Complexity; Computation; Computational linguistics; Image processing; Language; Language modeling; Linguistics; Machine translation; Memory; Modelling; Neighborhoods; Neural networks; Source code; Sparsity; Transformers