Audio-Visual Speech Recognition with Scattering Operators

One or more non-transitory computer readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

• receiving audiovisual input comprising audio input and video input associated with a subject;
• extracting visual features from the video input;
• applying a scattering operation to the extracted visual features to generate a vector of scattering coefficients in a first dimensional space;
• providing the vector of scattering coefficients as input to a first neural network for visual processing;
• providing the audio input to a second neural network for audio processing;
• combining a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input;
• providing the fused audiovisual feature vector to a third neural network for audiovisual processing;
• and
• generating, using the third neural network, a prediction regarding a speech status of the subject based on the fused audiovisual feature vector.

An apparatus comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
• receive audiovisual input comprising audio input and video input associated with a subject;
• extract visual features from the video input;
• apply a scattering operation to the extracted visual features to generate a vector of scattering coefficients;
• provide the vector of scattering coefficients as input to a first neural network for visual processing;
• provide the audio input to a second neural network for audio processing;
• combine a first output of the first neural network with a second output of the second neural network to generate a fused audiovisual feature vector based on the audiovisual input;
• provide the fused audiovisual feature vector to a third neural network for audiovisual processing;
• and
• generate, and using the third neural network, a first prediction regarding a speech status of the subject based on the fused audiovisual feature vector.