US 12,731,587 B2Granted

Visual speech recognition for digital videos utilizing generative adversarial learning

AssigneeAdobe Inc.

InventorsYaman Kumar, Balaji Krishnamurthy

Life of this patent

Estimating the term…

Loading patent…

Abstract

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that recognize speech from a digital video utilizing an unsupervised machine learning model, such as a generative adversarial neural network (GAN) model. In one or more implementations, the disclosed systems utilize an image encoder to generate self-supervised deep visual speech representations from frames of an unlabeled (or unannotated) digital video. Subsequently, in one or more embodiments, the disclosed systems generate viseme sequences from the deep visual speech representations (e.g., via segmented visemic speech representations from clusters of the deep visual speech representations) utilizing the adversarially trained GAN model. Indeed, in some instances, the disclosed systems decode the viseme sequences belonging to the digital video to generate an electronic transcription and/or digital audio for the digital video.

Claims20

Dependent claims

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to:

train a generative adversarial neural network to generate viseme sequences utilizing visual training data by:
generating, utilizing parameters of the generative adversarial neural network, viseme sequence predictions from the visual training data;
generating, utilizing a viseme sequence discriminator neural network, authenticity predictions from the viseme sequence predictions;
generating a segment smoothness loss, wherein the segment smoothness loss penalizes sequential changes of structural mouth movements indicated by visemic speech representations of the visual training data;
and
generating modified parameters of the generative adversarial neural network based on the authenticity predictions and the segment smoothness loss;
generate, utilizing a transformer-based image encoder, a set of deep visual speech representations for a set of digital video frames of a digital video by generating a deep visual visemic speech representation that represents viseme features visually portrayed in a digital video frame from the set of digital video frames, wherein a viseme feature from the viseme features corresponds to a visual viseme relating to multiple audible phonemes;
generate, utilizing the modified parameters of the generative adversarial neural network, a viseme sequence from the set of deep visual speech representations;
and
decode, utilizing a language decoder, the viseme sequence to generate an electronic transcription that represents speech portrayed within the digital video.

7 more elements

A system comprising:

one or more memory devices comprising a digital video, a transformer-based image encoder, and a generative adversarial neural network;
and
one or more processors configured to cause the system to:
generate, utilizing the transformer-based image encoder, a set of deep visual speech representations for a set of digital video frames of the digital video by generating a deep visual visemic speech representation that represents viseme features visually portrayed in a digital video frame from the set of digital video frames, wherein a viseme feature from the viseme features corresponds to a visual viseme relating to multiple audible phonemes;
generate deep visual speech representation clusters from the set of deep visual speech representations;
determine segmented visemic speech representations from the deep visual speech representation clusters;
and
generate a viseme sequence that represents speech portrayed within the digital video from the segmented visemic speech representations by utilizing the generative adversarial neural network trained utilizing a viseme sequence discriminator neural network that generates authenticity predictions from visual training data comprising the visual viseme relating to the multiple audible phonemes.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/650,020Filed 4 Feb 2022
Published: US 2023/0252993 A110 Aug 2023
Granted: 8 Sep 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification15 codes

G10L 15/25
Speech recognition using non-acoustical features: using position of the lips, movement of the lips or face analysis
G06F 18/23213
Using statistics or function optimisation, e.g. modelling of probability density functions: with fixed number of clusters, e.g. K-means clustering
G06N 3/02
Computing arrangements based on biological models: neural networks
G06T 9/002
Image coding: using neural networks