US 9,418,650 B2Granted

Training speech recognition using captions

AssigneeVerizon Patent and Licensing Inc.

InventorsSujeeth S. Bharadwaj, Suri B. Medapati

Life of this patent

Estimating the term…

Loading patent…

Abstract

In embodiments, apparatuses, methods and storage media are described that are associated with training adaptive speech recognition systems (“ASR”) using audio and text obtained from captioned video. In various embodiments, the audio and caption may be aligned for identification, such as according to a start and end time associated with a caption, and the alignment may be adjusted to better fit audio to a given caption. In various embodiments, the aligned audio and caption may then be used for training if an error value associated with the audio and caption demonstrates that the audio and caption will aid in training the ASR. In various embodiments, filters may be used on audio and text prior to training. Such filters may be used to exclude potential training audio and text based on filter criteria. Other embodiments may be described and claimed.

Claims25

Dependent claims

An apparatus, comprising:

a memory configured to store instructions;
and
a processor configured to execute the instructions to:
receive multimedia content, wherein the multimedia content include video, audio, and captions, wherein the captions identify a textual representation of the audio, and wherein the multimedia content relates to dialog by a first speaker and a second speaker;
perform image processing on the video to identify an age group of the first speaker and a gender of the first speaker, wherein the second speaker is not associated with at least one of the age group or the gender;
determine, based on the image processing, a first portion of the video presenting a first portion of the dialog by the first speaker and a second portion of the video presenting a second portion of the dialog by the second speaker, wherein a first portion of the captions is associated with the first portion of the dialog and a second portion of the captions is associated with the second portion of the dialog;
identify a first portion of the audio associated with the first portion of the video, and a second portion of the audio associated with the second portion of the video;
filter the multimedia content to extract the first portion of the audio and the first portion of the captions and to remove the second portion of the audio and the second portion of the captions;
and
train an adaptive speech recognition system (ASR) with respect to the age group of the first speaker and the gender of the first speaker, wherein the processor, when training the ASR with respect to the age group of the first speaker and the gender of the first speaker, is further configured to:
use the untrained ASR to perform a speech-to-text conversion of the first portion of the audio;
calculate an error value based on comparing the first portion of the captions and results from the speech-to-text conversion using the untrained ASR;
and
use the first portion of the audio and the first portion of the captions to train the ASR when the error value is between a first threshold and a second threshold that is greater than the first threshold, wherein the first portion of the audio and the first portion of the captions are not used to train the ASR with respect to the age group and the gender of the first speaker when the error value is below the first threshold or greater than the second threshold.

11 more elements

A method comprising:

receiving, by a processor, multimedia content, wherein the multimedia content include video, audio, and captions, wherein the captions identify a textual representation of the audio, and wherein the multimedia content relates to dialog by a first speaker and a second speaker;
performing, by the processor, image processing on the video to identify an age group of the first speaker and a gender of the first speaker, wherein the second speaker is not associated with at least one of the age group or the gender;
determining, by the processor and based on the image processing, a first portion of the video presenting a first portion of the dialog by the first speaker and a second portion of the video presenting a second portion of the dialog by the second speaker, wherein a first portion of the captions is associated with the first portion of the dialog and a second portion of the captions is associated with the second portion of the dialog;
identifying, by the processor, a first portion of the audio associated with the first portion of the video, and a second portion of the audio associated with the second portion of the video;
filtering, by the processor, the multimedia content to extract the first portion of the audio and the first portion of the captions and to remove the second portion of the audio and the second portion of the captions, wherein filtering the multimedia content includes:
determining a confidence value associated with an alignment between the first portion of the audio and the first portion of the captions;
determining whether the confidence value satisfies a confidence threshold;
and
adjusting, when the confidence value does not satisfy the confidence threshold, at least one of at least one of a start time for the first portion of the audio or the end time for the first portion of the audio;
and
training, by the processor, an adaptive speech recognition system (ASR) with respect to the age group of the first speaker and the gender of the first speaker using the first portion of the audio and the first portion of the captions, wherein training the ASR with respect to the age group and the gender of the first speaker further includes:
using the untrained ASR to perform a speech-to-text conversion of the first portion of the audio;
calculating an error value based on comparing the first portion of the captions and results from the speech-to-text conversion using the untrained ASR;
and
using the first portion of the audio and the first portion of the captions to train the ASR with respect to the age group and the gender of the first speaker when the error value is between a first threshold and a second threshold, wherein the first threshold is greater than the second threshold, and wherein the first portion of the audio and the first portion of the captions are not used to train the ASR with respect to the age group and the gender of the first speaker when the error value is above the first threshold or below than the second threshold.

12 more elements

A non-transitory computer-readable medium to store instructions, the instructions comprising:

one or more instructions that, when executed by a processor, cause the processor to:
receive multimedia content that includes video, audio, and captions identifying a textual representation of the audio, wherein the multimedia content relates to dialog by a first speaker and a second speaker;
perform image processing on the video to identify an age group of the first speaker and a gender of the first speaker, wherein the second speaker is not associated with at least one of the age group or the gender;
determine, based on the image processing, a first portion of the video presenting a first portion of the dialog by the first speaker and a second portion of the video presenting a second portion of the dialog by the second speaker, wherein a first portion of the captions is associated with the first portion of the dialog and a second portion of the captions is associated with the second portion of the dialog;
identify a first portion of the audio associated with the first portion of the video, and a second portion of the audio associated with the second portion of the video, wherein the one or more instructions further cause the processor, when identifying the portion of audio associated with the captions of the video, to:
determine a first confidence value associated with an alignment between a first part of the audio associated with the first speaker and a corresponding first part of the captions;
determine a second confidence value associated with an alignment between a second part of the audio associated with the first speaker and a corresponding second part of the captions;
and
select, as the first portion of the audio to be used for training the ASR with respect to the age group and the gender of the first speaker, the first part of the audio or the second part of the audio based on a comparison of the first confidence value and the second confidence value;
filter the multimedia content to extract the first portion of the audio and the first portion of the captions and to remove the second portion of the audio and the second portion of the captions;
and
train an adaptive speech recognition system (ASR) with respect to the age group of the first speaker and the gender of the first speaker using the first portion of the audio and the first portion of the captions, wherein the one or more instructions further cause the processor, when training the ASR, to:
use the untrained ASR to perform a speech-to-text conversion of the first portion of the audio;
calculate an error value based on comparing the first portion of the captions and results from the speech-to-text conversion using the untrained ASR;
and
use the first portion of the audio and the first portion of the captions to train the ASR with respect to the age group and the gender of the first speaker when the error value is between a first threshold and a second threshold, wherein the first threshold is greater than the second threshold, and wherein the first portion of the audio and the first portion of the captions are not used to train the ASR with respect to the age group and the gender of the first speaker when the error value is above the first threshold or below than the second threshold.

13 more elements

Dependencies are inferred from the claim text.