Augmenting attentioned-based neural networks to selectively attend to past inputs
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for...
Saved in:
Main Authors | , , |
---|---|
Format | Patent |
Language | English |
Published |
28.11.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input. |
---|---|
AbstractList | Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input that is a sequence to generate a network output. In one aspect, one of the methods includes, for each particular sequence of layer inputs: for each attention layer in the neural network: maintaining episodic memory data; maintaining compressed memory data; receiving a layer input to be processed by the attention layer; and applying an attention mechanism over (i) the compressed representation in the compressed memory data for the layer, (ii) the hidden states in the episodic memory data for the layer, and (iii) the respective hidden state at each of the plurality of input positions in the particular network input to generate a respective activation for each input position in the layer input. |
Author | Potapenko, Anna Lillicrap, Timothy Paul Rae, Jack William |
Author_xml | – fullname: Rae, Jack William – fullname: Potapenko, Anna – fullname: Lillicrap, Timothy Paul |
BookMark | eNrjYmDJy89L5WQIdCxNz03NK8nMS1dILCkBsYDiKbpJicWpKQp5qaVFiTlAqqQ8vyi7WKEkX6E4NSc1uSSzLDWnEqIhBSRakFhcopCZV1BaUszDwJqWmFOcyguluRkU3VxDnD10Uwvy41OLCxKTU4HmxYcGGxpaGFlaWJg4GRkTowYAflE6hA |
ContentType | Patent |
DBID | EVB |
DatabaseName | esp@cenet |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: EVB name: esp@cenet url: http://worldwide.espacenet.com/singleLineSearch?locale=en_EP sourceTypes: Open Access Repository |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Medicine Chemistry Sciences Physics |
ExternalDocumentID | US11829884B2 |
GroupedDBID | EVB |
ID | FETCH-epo_espacenet_US11829884B23 |
IEDL.DBID | EVB |
IngestDate | Fri Jul 19 12:56:36 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-epo_espacenet_US11829884B23 |
Notes | Application Number: US202017033396 |
OpenAccessLink | https://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20231128&DB=EPODOC&CC=US&NR=11829884B2 |
ParticipantIDs | epo_espacenet_US11829884B2 |
PublicationCentury | 2000 |
PublicationDate | 20231128 |
PublicationDateYYYYMMDD | 2023-11-28 |
PublicationDate_xml | – month: 11 year: 2023 text: 20231128 day: 28 |
PublicationDecade | 2020 |
PublicationYear | 2023 |
RelatedCompanies | DeepMind Technologies Limited |
RelatedCompanies_xml | – name: DeepMind Technologies Limited |
Score | 3.5033262 |
Snippet | Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input... |
SourceID | epo |
SourceType | Open Access Repository |
SubjectTerms | CALCULATING COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS COMPUTING COUNTING ELECTRIC DIGITAL DATA PROCESSING PHYSICS |
Title | Augmenting attentioned-based neural networks to selectively attend to past inputs |
URI | https://worldwide.espacenet.com/publicationDetails/biblio?FT=D&date=20231128&DB=EPODOC&locale=&CC=US&NR=11829884B2 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1ZS8NAEB5KPd80KloPIkjegj1y9SGIuShCD20jfSu5WiolDe4G8d87s6bWF30KTNjdZGBmvmy--RbgTk8QBFlmrOrEbNfSrKnG2txQNdqARESiJ6KLvz8weqH2NNWnNXjb9MIIndAPIY6IEZVgvHORr4vtJpYnuJXsPl6iaf0QTGxPqb6OEaxgvlU8x_ZHQ2_oKq5rh2Nl8GITju5aluZgut4hGE06-_6rQ10pxe-SEhzB7ghny_kx1LJcggN3c_KaBPv96oe3BHuCoZkwNFZRyE7g-bFcCJZPvpBJHlMQFrNUpYqUyqRQGa3wIvjdTOZrmYnDbjCvrT6_B6RkLSLG5WVelJydwm3gT9yeik85-3HJLBxvX6hzBvUclzkHuRsZhtUyzLbRibSE1N3MZiuJMYgzkt1JL6Dx9zyN_25ewiG5l7rw2tYV1Pl7mV1jOebxjfDjF--kjww |
link.rule.ids | 230,309,786,891,25594,76906 |
linkProvider | European Patent Office |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT8JAEJ4QfOBNqwbxVRPTWyOPvjg0RloIKgUUMNyaviQYUhp3G-O_d2YF8aKnJtPsbjvJzHzdfvMtwLUeIQiyzFDVidmuxUlVDbVXQ9VoAxIRiR6JLn6vb3Qn2sNUnxbgbd0LI3RCP4Q4IkZUhPHORb7ONptYruBWsptwjqblbWdsu8rq6xjBCuZbxW3Z7eHAHTiK49iTkdJ_tglHNy1La2G63jJJnZeg00uLulKy3yWlsw_bQ5wt5QdQSFIJSs765DUJdr3VD28JdgRDM2JoXEUhO4Snu3wmWD7pTCZ5TEFYTGKVKlIsk0JlsMCL4HczmS9lJg67wby2-PweEJM1CxiX52mWc3YEV5322Omq-JT-j0v8yWjzQo1jKKa4TBnkZmAYVs0w60Yj0CJSdzOrtSjEIE5Idic-gcrf81T-u3kJpe7Y6_m9-_7jKeyRq6kjr26dQZG_58k5lmYeXgiffgHJZpH5 |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Apatent&rft.title=Augmenting+attentioned-based+neural+networks+to+selectively+attend+to+past+inputs&rft.inventor=Rae%2C+Jack+William&rft.inventor=Potapenko%2C+Anna&rft.inventor=Lillicrap%2C+Timothy+Paul&rft.date=2023-11-28&rft.externalDBID=B2&rft.externalDocID=US11829884B2 |