Visual speech recognition by phoneme prediction

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for visual speech recognition, the operations comprising:

• receiving a video comprising a plurality of video frames, wherein each video frame depicts a pair of lips;
• processing the video using a visual speech recognition neural network in accordance with current values of visual speech recognition neural network parameters to generate, for each output position in an output sequence, a respective output score for each token in a vocabulary of possible tokens,
• wherein the visual speech recognition neural network comprises:
• (i) a three-dimensional (3D) convolutional subnetwork comprising a sequence of multiple volumetric convolutional neural network layers, and (ii) a temporal subnetwork;
• wherein the 3D convolutional subnetwork processes the plurality of video frames depicting the pair of lips using a plurality of three-dimensional (3D) convolutional filters of the sequence of multiple volumetric convolutional neural network layers to generate a respective spatio-temporal feature tensor for each video frame of the plurality of video frames depicting the pair of lips;
• wherein the temporal subnetwork processes the spatio-temporal feature tensors corresponding to the video frames depicting the pair of lips to generate, for each output position in the output sequence, the respective output score for each token in the vocabulary of possible tokens;
• wherein the vocabulary of possible tokens comprises a plurality of phonemes;
• and
• determining a sequence of words expressed by the pair of lips depicted in the video using, for each output position in the output sequence, the respective output score for each token in the vocabulary of possible tokens.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for visual speech recognition, the operations comprising:

• receiving a video comprising a plurality of video frames, wherein each video frame depicts a pair of lips;
• processing the video using a visual speech recognition neural network in accordance with current values of visual speech recognition neural network parameters to generate, for each output position in an output sequence, a respective output score for each token in a vocabulary of possible tokens,
• wherein the visual speech recognition neural network comprises:
• (i) a three-dimensional (3D) convolutional subnetwork comprising a sequence of multiple volumetric convolutional neural network layers, and (ii) a temporal subnetwork;
• wherein the 3D convolutional subnetwork processes the plurality of video frames depicting the pair of lips using a plurality of three-dimensional (3D) convolutional filters of the sequence of multiple volumetric convolutional neural network layers to generate a respective spatio-temporal feature tensor for each video frame of the plurality of video frames depicting the pair of lips;
• wherein the temporal subnetwork processes the spatio-temporal feature tensors corresponding to the video frames depicting the pair of lips to generate, for each output position in the output sequence, the respective output score for each token in the vocabulary of possible tokens;
• wherein the vocabulary of possible tokens comprises a plurality of phonemes;
• and
• determining a sequence of words expressed by the pair of lips depicted in the video using, for each output position in the output sequence, the respective output score for each token in the vocabulary of possible tokens.