# US 2022/0044668 A1 · Translating between spoken languages with emotion in audio and video media streams

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044668 A1
- Document ID: US20220044668A1
- Status: Published application
- Kind: A1
- Application number: 17/509,401
- Filed: 2021-10-25
- Published: 2022-02-10
- Earliest claimed: 2018-10-04
- Assignees: Rovi Guides, Inc.
- Applicants: Rovi Guides, Inc.
- Inventors: Vijay Kumar; Rajendran Pichaimurthy; Madhusudhan Seetharam
- Web page: https://patentgrep.com/patent/US20220044668A1

## CPC

- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/4856: {for language selection, e.g. for the menu or subtitles}
- G06F 40/30: Semantic analysis
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 15/07: to the speaker
- G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
- G10L 25/63: for estimating an emotional state
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/4398: {involving reformatting operations of audio signals (details of audio signal transcoding G10L19/173)}
- H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- G10L 17/00: Speaker identification or verification techniques
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods are described herein for generating alternate audio for a media stream. The media system receives media that is requested by the user. The media comprises a video and audio. The audio includes words spoken in a first language. The media system stores the received media in a buffer as it is received. The media system separates the audio from the buffered media and determines an emotional state expressed by spoken words of the first language. The media system translates the words spoken in the first language into words spoken in a second language. Using the translated words of the second language, the media system synthesizes speech having the emotional state previously determined. The media system then retrieves the video of the received media from the buffer and synchronizes the synthesized speech with the video to generate the media content in a second language.

## Claims

### Claim at position 1

1-27. (canceled)

### Claim 28

A method comprising:
accessing a media asset, the media asset featuring a speaker;
receiving a request to translate the media asset;
searching for another media asset featuring the speaker;
extracting a voice sample from the another media asset featuring the speaker;
calculating vocal characteristics based on the voice sample from the another media asset featuring the speaker; and
generating a translation of the media asset using the calculated vocal characteristics.

### Claim 29 (depends on 28)

The method of claim 28, wherein the accessing the media asset comprises receiving the media asset comprising audio, wherein the audio comprises a first plurality of spoken words, the method further comprising:
retrieving audio from the received media asset;
determining an emotional state expressed by the first plurality of spoken words based on a first set of characteristics associated with the first plurality of spoken words;
identifying an identifier of a speaker that produces the first plurality of spoken words based on metadata associated with the media asset;
identifying characteristics of the identified speaker that produces the first plurality of spoken words by:
searching, based on the identifier, for another media asset featuring the speaker;
the extracting the voice sample from the another media asset featuring the speaker; and
the calculating the vocal characteristics based on the voice sample from the another media asset featuring the speaker; and
converting the first plurality of spoken words into a second plurality of spoken words,
wherein the generating the translation comprises generating a converted audio, based on the second plurality of spoken words and the characteristics of the speaker that produces the first plurality of spoken words, wherein the converted audio comprises a second set of characteristics associated with the determined emotional state.

### Claim 30 (depends on 29)

The method of claim 29, wherein converting the first plurality of spoken words into the second plurality of spoken words comprises:
translating the first plurality of spoken words of a first language into the second plurality of spoken words of a second language;
transcribing the first plurality of spoken words in the first language;
translating the transcribed words of the first language into words of the second language; and
wherein generating the translated audio comprises synthesizing a speech from the translated words of the second language.

### Claim 31 (depends on 30)

The method of claim 30, wherein translating the transcribed words of the first language into words of the second language comprises:
retrieving, from a language translation database, rules of grammar of the first language;
determining, using the rules of grammar of the first language, parts of speech of the translated words of the first language;
retrieving, from the language translation database, rules of grammar of the second language; and
translating the translated words of the first language into words of the second language based on the parts of speech assigned to the transcribed words of the first language and rules of grammar of the second language.

### Claim 32 (depends on 29)

The method of claim 29, wherein determining the emotional state expressed by the first plurality of spoken words comprises:
determining whether the first plurality of spoken words contain emotion identifiers;
in response to determining that the first plurality of spoken words contain emotion identifiers:
comparing the emotion identifier to a plurality of emotional identifiers stored in a database; and
determining the emotional state expressed by the first plurality of spoken words based on matches resulting from the comparison.

### Claim 33 (depends on 29)

The method of claim 29, wherein determining the emotional state expressed by the first plurality of spoken words further comprises:
determining the first set of characteristics associated with the first plurality of spoken words;
comparing the first set of characteristics to a particular set of characteristics stored in a language conversion database; and
determining the emotional state expressed by the first plurality of spoken words based on matches resulting from the comparison.

### Claim 34 (depends on 33)

The method of claim 33, further comprising:
retrieving video from the received media asset; and
generating for output the retrieved video and the converted audio, wherein the converted audio is output with the video instead of the first plurality of spoken words, wherein generating the converted audio from the second plurality of spoken words comprises:
determining, based on the first set of characteristics, a gender of the speaker that produces the first plurality of spoken words;
retrieving, from a language conversion database, the second set of characteristics based on the determined gender of the speaker; and
synthesizing the speech using the second set of characteristics.

### Claim 35 (depends on 33)

The method of claim 33, wherein generating the converted audio from the second plurality of spoken words comprises:
determining, based on the first set of characteristics, an ethnicity of the speaker that produces the first plurality of spoken words;
retrieving, from a language conversion database, the second set of characteristics based on the determined ethnicity of the speaker; and
synthesizing the speech using the second set of characteristics.

### Claim 36 (depends on 29)

The method of claim 29, wherein generating the converted audio from the second plurality of spoken words comprises:
determining an identity of the person that produces the first plurality of spoken words;
retrieving, from a language conversion database, a plurality of voice samples of the speaker;
calculating a vocal fingerprint of the speaker using the plurality of voice samples; and
synthesizing speech based on the calculated vocal fingerprint.

### Claim 37 (depends on 29)

The method of claim 29, wherein:
the method comprises receiving an input to generate alternate audio for a media asset stream; and
determining the emotional state, converting the first plurality of spoken words, synthesizing speech from the second plurality of spoken words, and generating, the retrieved video with the synthesized speech for display are performed in response to the received user input.

### Claim 38

A system comprising control circuitry configured to:
access a media asset, the media asset featuring a speaker;
receive a request to translate the media asset;
search for another media asset featuring the speaker;
extract a voice sample from the another media asset featuring the speaker;
calculate vocal characteristics based on the voice sample from the another media asset featuring the speaker; and
generate a translation of the media asset using the calculated vocal characteristics.

### Claim 39 (depends on 38)

The system of claim 38, wherein the control circuitry configured to access the media asset comprises control circuitry configured to receive the media asset comprising audio, wherein the audio comprises a first plurality of spoken words, the control circuitry further configured to:
retrieve audio from the received media asset:
determine an emotional state expressed by the first plurality of spoken words based on a first set of characteristics associated with the first plurality of spoken words;
identify an identifier of a speaker that produces the first plurality of spoken words based on metadata associated with the media asset;
identify characteristics of the identified speaker that produces the first plurality of spoken words by:
searching, based on the identifier, for another media asset featuring the speaker;
the extracting the voice sample from the another media asset featuring the speaker; and
the calculating the vocal characteristics based on the voice sample from the another media asset featuring the speaker; and
convert the first plurality of spoken words into a second plurality of spoken words,
wherein the control circuitry configured to generate the translation comprises control circuitry configured to generate a converted audio, based on the second plurality of spoken words and the characteristics of the speaker that produces the first plurality of spoken words, wherein the converted audio comprises a second set of characteristics associated with the determined emotional state.

### Claim 40 (depends on 39)

The system of claim 39, wherein the control circuitry, when converting the first plurality of spoken words into the second plurality of spoken words is further configured to:
translate the first plurality of spoken words of a first language into the second plurality of spoken words of a second language;
transcribe the first plurality of spoken words in the first language;
translate the transcribed words of the first language into words of the second language; and
wherein generating the translated audio comprises synthesizing a speech from the translated words of the second language.

### Claim 41 (depends on 40)

The system of claim 40, wherein the control circuitry, when translating the transcribed words of the first language into words of the second language is further configured to:
retrieve, from a language translation database, rules of grammar of the first language;
determine, using the rules of grammar of the first language, parts of speech of the translated words of the first language;
retrieve, from the language translation database, rules of grammar of the second language; and
translate the translated words of the first language into words of the second language based on the parts of speech assigned to the transcribed words of the first language and rules of grammar of the second language.

### Claim 42 (depends on 39)

The system of claim 39, wherein the control circuitry, when determining the emotional state expressed by the first plurality of spoken words is further configured to:
determine whether the first plurality of spoken words contain emotion identifiers; and
in response to determining that the first plurality of spoken words contain emotion identifiers:
compare the emotion identifier to a plurality of emotional identifiers stored in a database; and
determine the emotional state expressed by the first plurality of spoken words based on matches resulting from the comparison.

### Claim 43 (depends on 39)

The system of claim 39, wherein control circuitry, when determining, the emotional state expressed by the first plurality of spoken words is further configured to:
determine the first set of characteristics associated with the first plurality of spoken words;
compare the first set of characteristics to a particular set of characteristics stored in a language conversion database; and
determine the emotional state expressed by the first plurality of spoken words based on matches resulting from the comparison.

### Claim 44 (depends on 43)

The system of claim 43, wherein the control circuitry is further configured to:
retrieve video from the received media asset; and
generate for output the retrieved video and the converted audio, wherein the converted audio is output with the video instead of the first plurality of spoken words;
wherein the control circuitry, when generating the converted audio from the second plurality of spoken words is further configured to:
determine, based on the first set of characteristics, a gender of the speaker that produces the first plurality of spoken words;
retrieve, from a language conversion database, the second set of characteristics based on the determined gender of the speaker; and
synthesize the speech using the second set of characteristics.

### Claim 45 (depends on 43)

The system of claim 43, wherein the control circuitry, when generating the converted audio from the second plurality of spoken words is further configured to:
determine, based on the first set of characteristics, an ethnicity of a speaker that produces the first plurality of spoken words;
retrieve, from a language conversion database, the second set of characteristics based on the determined ethnicity of the speaker; and
synthesize the speech using the second set of characteristics.

### Claim 46 (depends on 39)

The system of claim 39, wherein the control circuitry, when generating the converted audio from the second plurality of spoken words is further configured to:
determine an identity of the speaker that produces the first plurality of spoken words;
retrieve, from a language conversion database, a plurality of voice samples of the speaker;
calculate a vocal fingerprint of the speaker using the plurality of voice samples; and
synthesize speech based on the calculated vocal fingerprint.

### Claim 47 (depends on 39)

The system of claim 39, wherein the control circuitry is further configured to:
receive an input to generate alternate audio for a media asset stream; and determine the emotional state, convert the first plurality of spoken words, synthesize speech from the second plurality of spoken words, and generate, the retrieved video with the synthesized speech for display are performed in response to the received user input.

### Claim 48

A non-transitory computer readable medium comprising instructions that when executed cause the control circuitry to:
accessing a media asset, the media asset featuring a speaker;
receiving a request to translate the media asset;
searching for another media asset featuring the speaker;
extracting a voice sample from the another media asset featuring the speaker;
calculating vocal characteristics based on the voice sample from the another media asset featuring the speaker; and
generating a translation of the media asset using the calculated vocal characteristics.
