US 11,900,919 B2 · End-to-end multi-speaker audio-visual automatic speech recognition

Abstract

An audio-visual automated speech recognition model for transcribing speech from audio-visual data includes an encoder frontend and a decoder. The encoder includes an attention mechanism configured to receive an audio track of the audio-visual data and a video portion of the audio-visual data. The video portion of the audio-visual data includes a plurality of video face tracks each associated with a face of a respective person. For each video face track of the plurality of video face tracks, the attention mechanism is configured to determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track includes a speaking face of the audio track. The decoder is configured to process the audio track and the video face track of the plurality of video face tracks associated with the highest confidence score to determine a speech recognition result of the audio track.

Claims20

Dependent claims

An audio-visual automated speech recognition (AV-ASR) model for transcribing speech from audio-visual data, the AV-ASR model comprising:

an encoder configured to:
receive a sequence of acoustic feature vectors for an audio track and a plurality of video face tracks, each video face track of the plurality of video face tracks is associated with a face of a respective person;
and
for each corresponding acoustic feature vector in the sequence of acoustic feature vectors:
determine a respective attention-weighted visual feature vector that soft-selects a particular one of the video face tracks that is most likely to include the face of the respective person as a speaking face of the corresponding acoustic feature vector;
and
generate a combined feature vector by concatenating the respective attention-weighted visual feature vector with the corresponding acoustic feature vector;
and
a decoder configured to process the combined feature vector generated for each corresponding acoustic feature vector in the sequence of acoustic feature vectors to determine a speech recognition result of the audio track.

6 more elements

Dependencies are inferred from the claim text.

End-to-end multi-speaker audio-visual automatic speech recognition

Life of this patent

End-to-end multi-speaker audio-visual automatic speech recognition

Life of this patent

Abstract

Claims20