US 10,945,041 B1Granted

Language-agnostic subtitle drift detection and localization

AssigneeAmazon Technologies, Inc.

InventorsTamojit Chatterjee, Mayank Sharma, Muhammad Raffay Hamid, Sandeep Joshi

Life of this patent

Estimating the term…

Loading patent…

Abstract

Devices, systems, and methods are provided for language-agnostic subtitle drift detection and localization. A method may include extracting audio from video, dividing the audio into overlapping blocks, and determining the probabilities of overlapping portions of the blocks, the probabilities indicating a presence of voice data represented by the audio in the blocks. The method may generate machine blocks using overlapping portions of blocks where voice data is present, and may map the machine blocks to corresponding blocks indicating that subtitles are available for the video. For mapped blocks, the method may include determining features such as when subtitles are available without voice audio, when voice audio is available without subtitles, and when voice audio and subtitles both are available. Using the features, the method may include determining the probability that the video includes subtitle drift, and the method may include analyzing the video to localize where the subtitle drift occurs.

Claims20

Dependent claims

A method, comprising:

generating, by at least one processor of a system, a first block of audio for a video and a second block of audio for the video, the first block and the second block having a first duration;
determining, by the at least one processor, a first probability indicating that the first block includes audio data representative of a voice;
determining, by the at least one processor, a second probability indicating that the second block includes audio data representative of a voice;
determining, by the at least one processor, an overlapping of the first block and the second block;
generating, by the at least one processor and based on the overlapping, a third block of audio for the video and a fourth block of audio for the video, the third block and the fourth block having a second duration less than the first duration;
determining, by the at least one processor, a third probability indicating that the third block includes audio data representative of a voice;
determining, by the at least one processor, a fourth probability indicating that the fourth block includes audio data representative of a voice;
generating, by the at least one processor and based on the third probability and the fourth probability, a fifth block of audio for the video, the fifth block comprising the third block and the fourth block, and the fifth block associated with a first language;
determining, by the at least one processor, a mapping between the fifth block and a sixth block of audio for the video, the sixth block indicating that subtitles occur between a start time of the sixth block and an end time of the sixth block, the start time of the sixth block occurring before a start time of the fifth block, the end time of the sixth block occurring before an end time of the fifth block, and the sixth block associated with a second language different than the first language;
determining, by the at least one processor, a start gap between the start time of the sixth block and the start time of the fifth block, an end gap between the end time of the sixth block and the end time of the fifth block, and a percentage of time by which the fifth block is overlapped by the sixth block;
determining, by the at least one processor and based on the start gap, the end gap, and the percentage of time, a fifth probability indicating that the video includes subtitle drift;
generating, by the at least one processor, a seventh block of audio for the video, the seventh block comprising the first block and the second block, and the seventh block having a third duration greater than the first duration and less than a third duration of the video;
and
determining, by the at least one processor and based on the fifth probability and the subtitle drift associated with the seventh block, that the seventh block includes subtitle drift.

11 more elements

A method, comprising:

generating, by at least one processor of a system, a first block of audio for a video and a second block of audio for the video, the first block and the second block having a first duration;
determining, by the at least one processor, a first probability indicating that the first block includes audio data representative of a voice;
determining, by the at least one processor, a second probability indicating that the second block includes audio data representative of a voice;
generating, by the at least one processor and based on an overlapped portion of the first block and of the second block, a third block of audio for the video and a fourth block of audio for the video, the third block and the fourth block having a second duration less than the first duration;
generating, by the at least one processor, a fifth block of audio for the video, the fifth block comprising the third block and the fourth block;
determining, by the at least one processor, a mapping between the fifth block and a sixth block of audio for the video, the sixth block indicating that subtitles occur during between a start time of the sixth block and an end time of the sixth block;
determining, by the at least one processor, a start gap between a start time of the sixth block and a start time of the fifth block, an end gap between associated with an end time of the fifth block, and a percentage of time by which the fifth block is overlapped by the sixth block;
determining, by the at least one processor and based on the start gap, the end gap, and the percentage of time, a third probability indicating that the video includes subtitle drift;
generating, by the at least one processor, a seventh block of audio for the video;
and
determining, by the at least one processor, that the seventh block includes subtitle drift.

8 more elements

A system, comprising memory coupled to at least one processor, the at least one processor configured to:

generate a first block of audio for a video and a second block of audio for the video, the first block and the second block having a first duration;
determine a first probability indicating that the first block includes audio data representative of a voice;
determine a second probability indicating that the second block includes audio data representative of a voice;
generate, based on an overlapped portion of the first block and of the second block, a third block of audio for the video and a fourth block of audio for the video, the third block and the fourth block having a second duration less than the first duration;
generate a fifth block of audio for the video, the fifth block comprising the third block and the fourth block;
determine a mapping between the fifth block and a sixth block of audio for the video, the sixth block indicating that subtitles occur during between a start time of the sixth block and an end time of the sixth block;
determine a start gap between a start time of the sixth block and a start time of the fifth block, an end gap between associated with an end time of the fifth block, and a percentage of time by which the fifth block is overlapped by the sixth block;
determine, based on the start gap, the end gap, and the percentage of time, a fifth probability indicating that the video includes subtitle drift;
generate a seventh block of audio for the video;
and
determine that the seventh block includes subtitle drift.

8 more elements

Dependencies are inferred from the claim text.