Recurrent multimodal attention system based on expert gated networks

A method comprising:

• receiving a plurality of data streams associated with a plurality of input modalities, respectively;
• extracting, from each data stream of the plurality of data streams, features associated with a respective input modality of the plurality of input modalities;
• predicting a plurality of classes based on a plurality of recurrent neural networks (RNNs), respectively, each RNN of the plurality of RNNs being configured to predict the respective class based on the features associated a respective input modality of the plurality of input modalities;
• generating a set of weights for the plurality of input modalities based on a first neural network, the first neural network being trained to produce the set of weights based on the features associated with each input modality of the plurality of input modalities;
• and
• generating a weighted prediction based on the predicted classes and the set of weights.

A system comprising:

• a memory storing instructions;
• a processor coupled to the memory and configured to execute the instructions to cause the system to perform operations comprising:
• receiving a plurality of data streams associated with a plurality of input modalities, respectively;
• extracting, from each data stream of the plurality of data streams, features associated with a respective input modality of the plurality of input modalities;
• predicting a plurality of classes based on a plurality of recurrent neural networks (RNNs), respectively, each RNN of the plurality of RNNs being configured to predict the respective class based on the features associated with a respective input modality of the plurality of input modalities;
• generating a set of weights for the plurality of input modalities based on a first neural network, the first neural network being trained to produce the set of weights based on the features associated with each input modality of the plurality of input modalities;
• and
• generating a weighted prediction based on the plurality of classes and the set of weights.