Selecting isolated speaker signal by comparing text obtained from audio and video streams

An apparatus comprising:

• at least one processing device comprising a processor coupled to a memory;
• the at least one processing device being configured to implement the following steps:
• transforming a plurality of audio signals associated with at least one human speaker to respective ones of a plurality of first sets of predicted spoken words using at least one speech-to-text conversion model;
• transforming at least one video signal associated with the at least one human speaker to one or more second sets of predicted spoken words using at least one lip motion-to-text conversion model, wherein the one or more second sets of predicted spoken words are based at least in part on an analysis of at least one image associated with a respective human speaker;
• iteratively adjusting a steering vector associated with the plurality of audio signals to compare at least one of the plurality of first sets of predicted spoken words with at least one of the one or more second sets of predicted spoken words, wherein the iteratively adjusting the steering vector associated with the plurality of audio signals adjusts a position of the steering vector until the comparison, between the at least one of the plurality of first sets of predicted spoken words with the at least one of the one or more second sets of predicted spoken words, satisfies one or more designated criteria;
• and
• selecting an isolated human speaker audio signal associated with a particular one of the plurality of first sets of predicted spoken words and a particular one of the one or more second sets of predicted spoken words, wherein the selection is based at least in part on a result of the comparison.

A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:

• transforming a plurality of audio signals associated with at least one human speaker to respective ones of a plurality of first sets of predicted spoken words using at least one speech-to-text conversion model;
• transforming at least one video signal associated with the at least one human speaker to one or more second sets of predicted spoken words using at least one lip motion-to-text conversion model, wherein the one or more second sets of predicted spoken words are based at least in part on an analysis of at least one image associated with a respective human speaker;
• iteratively adjusting a steering vector associated with the plurality of audio signals to compare at least one of the plurality of first sets of predicted spoken words with at least one of the one or more second sets of predicted spoken words, wherein the iteratively adjusting the steering vector associated with the plurality of audio signals adjusts a position of the steering vector until the comparison, between the at least one of the plurality of first sets of predicted spoken words with the at least one of the one or more second sets of predicted spoken words, satisfies one or more designated criteria;
• and
• selecting an isolated human speaker audio signal associated with a particular one of the plurality of first sets of predicted spoken words and a particular one of the one or more second sets of predicted spoken words, wherein the selection is based at least in part on a result of the comparison.