Providing subtitle for video content in spoken language
AssigneeVoyagerX, Inc.
InventorHyeonsoo Oh
Life of this patent
Estimating the term…
Abstract
The present disclosure relates to systems and methods for providing subtitle for a video. The video's audio is transcribed to obtain caption text for the video. A first machine-trained model identifies sentences in the caption text. A second model identifies intra-sentence breaks with in the sentences identified using the first machine-trained model. Based on the identified sentences and intra-sentence breaks, one or more words in the caption text are grouped into a clip caption to be displayed for a corresponding clip of the video.
Claims17
A method of providing subtitle for a video, the method comprising:
- processing audio data of a video to generate a timed script in a first language which comprises a first sequence of words and a time stamp for each word of the first sequence of words;
- processing the first sequence of words to compute, using a first machine-trained model, a sentence-ending probability for each word of the first sequence of words;
- determining a first word of the first sequence as a first sentence-ending word based on the sentence-ending probability of the first word, which defines a first sentence that ends with the first word;
- processing the first sentence to compute, using a second machine-trained model, an intra-sentence break probability for at least one word of the first sentence;
- determining a second word of the first sentence as a clip-ending word based on the intra-sentence break probability of the second word, which defines a first clip text that ends with the second word, wherein defining of the first clip text further defines a first clip period that corresponds to the first clip text and ends at a time when the second word has been spoken in the video;
- and
- generating first language subtitle data comprising the first clip text and information indicative of the first clip period during which the first clip text is to be displayed as subtitle in the first language.
A system for providing subtitle for a video, the system comprising:
- at least one processor;
- and
- at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to:
- process audio data of a video to generate a timed script in a first language which comprises a first sequence of words and a time stamp for each word of the first sequence of words;
- process the first sequence of words to compute, using a first machine-trained model, a sentence-ending probability for each word of the first sequence of words;
- determine a first word of the first sequence as a first sentence-ending word based on the sentence-ending probability of the first word, which defines a first sentence that ends with the first word;
- process the first sentence to compute, using a second machine-trained model, an intra-sentence break probability for at least one word of the first sentence;
- determine a second word of the first sentence as a clip-ending word based on the intra-sentence break probability of the second word, which defines a first clip text that ends with the second word, wherein defining of the first clip text further defines a first clip that corresponds to the first clip text and ends at a time when the second word has been spoken in the video;
- and
- generate first language subtitle data comprising the first clip text and information indicative of a first clip period of the first clip during which the first clip text is to be displayed as subtitle in the first language.
Dependencies are inferred from the claim text.