TASK-SPECIFIC TEXT GENERATION BASED ON MULTIMODAL INPUTS

In one embodiment, a method includes accessing first sets of tokens associated with a desired task and one or more modalities associated with a context of the desired task, determining a second set of tokens for each of the one or more modalities using a classifier network associated with the modali...

Full description

Saved in:
Bibliographic Details
Main Authors WANG, Jue, PARIKH, Devi Niru, LIN, Xudong, TORRESANI, Lorenzo, BERTASIUS, Gediminas
Format Patent
LanguageEnglish
French
German
Published 20.07.2022
Subjects
CALCULATING
COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
COMPUTING
COUNTING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:In one embodiment, a method includes accessing first sets of tokens associated with a desired task and one or more modalities associated with a context of the desired task, determining a second set of tokens for each of the one or more modalities using a classifier network associated with the modality, generating a number of embedding vectors by mapping the first sets of tokens and the second set of tokens associated with each of the one or more modalities to an embedding space, and producing a sequence of words addressing the desired task by processing the number of embedding vectors with an encoder-decoder network.
Bibliography:Application Number: EP20210216259