Generating real-time audio dubbing for a video using a concurrent batch framework

A system comprising:

• a processing system having a processor;
• and
• a computer memory including instructions that, when executed by the processing system, cause the system to carry out operations comprising:
• segmenting audio in a first language from a video into a first audio segment and a second audio segment using an audio segmentation model on a first processing thread of a client device;
• generating a first translated text string in a second language from the first audio segment using a speech translation model on a second processing thread of the client device;
• generating a first translated audio segment in the second language from the first translated text string using a text-to-speech model corresponding to the second language on a third processing thread of the client device;
• concurrent with generating the first translated audio segment on the third processing thread, generating a second translated text string in the second language from the second audio segment using the speech translation model on the second processing thread of the client device;
• generating a second translated audio segment in the second language from the second translated text string using the text-to-speech model on the third processing thread of the client device;
• and
• providing the first translated audio segment with the video.

A computer-implemented method for generating real-time audio translations in one or more videos, comprising:

• segmenting audio in a first language from a video into a first audio segment and a second audio segment on a first processing thread of a client device;
• generating a first translated text string in a second language from the first audio segment using a speech translation model on a second processing thread of the client device before generating a second translated text string in the second language from the second audio segment using the speech translation model on the second processing thread;
• generating a first translated audio segment in the second language from the first translated text string using a text-to-speech model corresponding to the second language on a third processing thread of the client device concurrently with generating the second translated text string in the second language on the second processing thread;
• and
• providing the first translated audio segment with the video.