# US 11,443,737 B2 · Audio video translation into multiple languages for respective listeners

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,737 B2
- Document ID: US11443737B2
- Status: Granted
- Kind: B2
- Application number: 16/742,513
- Filed: 2020-01-14
- Published: 2021-07-15
- Granted: 2022-09-13
- Earliest claimed: 2020-01-14
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Bibhudendu Mohapatra
- Web page: https://patentgrep.com/patent/US11443737B2

## CPC

- G06F 40/42: Data-driven translation
- G10L 15/183: using context dependencies, e.g. language models
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/005: {Language recognition}
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04N 21/4126: {The peripheral being portable, e.g. PDAs or mobile phones}
- H04N 21/439: Processing of audio elementary streams
- H04N 21/4396: {by muting the audio signal}
- H04N 21/4398: {involving reformatting operations of audio signals (details of audio signal transcoding G10L19/173)}
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user}
- H04N 21/4856: {for language selection, e.g. for the menu or subtitles}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4518: {involving characteristics of one or more peripherals, e.g. peripheral type, software version, amount of memory available or display capabilities}
- H04N 21/4532: {involving end-user characteristics, e.g. viewer profile, preferences (monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An audio source such as a display device configured to present AV content can present the video and send the audio in different languages to the respective devices of different listeners. For example, a device/TV/source can send audio in different languages to connected headphones/smartglasses with speakers/devices/sink. Furthermore, machine learning may be employed both to recognize listeners and correlate them to likely languages and to mimic voices in the played-back audio. Or, the source AV display device may send language in only the selected language of the display device to each listener device, with each receiving listener device converting the audio to the preferred language of the respective listener on the fly.

## Claims

### Claim 1

An assembly, comprising:
at least one processor programmed to access instructions executable by the processor to:
receive at least audio in a first audio video (AV) content, the audio being in a first language;
for at least a first recipient device, provide a second language version of the audio for play on at least one speaker of the first recipient device while video of the AV content is presented on a display device different from the first recipient device, wherein the instructions are executable to translate the audio into the second language based at least in part on information output by at least one neural network executing face recognition of at least one input image.

### Claim 2 (depends on 1)

The assembly of claim 1, wherein the at least one processor is implemented in the display device, and the instructions are executable to translate the audio into the second language and send the audio in the second language to the first recipient device.

### Claim 3 (depends on 1)

The assembly of claim 1, wherein the second language is identified by at least one neural network.

### Claim 4 (depends on 1)

The assembly of claim 1, wherein the at least one processor is implemented in the display device, and the instructions are executable to access the audio in the second language from an advanced television systems committee (ATSC) data stream.

### Claim 5 (depends on 1)

The assembly of claim 1, wherein the at least one processor is implemented in the display device, and the instructions are executable to, for at least a second recipient device, provide a third language version of the audio for play on at least one speaker of the second recipient device while video of the AV content is presented on the display device, the display device being different from the first and second recipient devices.

### Claim 6 (depends on 1)

The assembly of claim 1, wherein the at least one processor is implemented in the first recipient device, and the instructions are executable to receive the audio in the first language and translate the audio to the second language as the video is presented on the display device.

### Claim 7

A system comprising:
at least a first display device configured to present video in an audio video (AV) stream and to present audio in the AV stream in a first language;
at least one processor configured to access the audio in at least a second language;
at least a first recipient device configured to play the audio in the second language while the display device presents the video; and
wherein the at least one processor is configured with instructions to identify the second language at least in part based on face recognition executed by at least one neural network.

### Claim 8 (depends on 7)

The system of claim 7, wherein the at least one processor is implemented in the display device, and the display device is configured to send the audio in the second language to the first recipient device and send the audio in a third language to a second recipient device.

### Claim 9 (depends on 7)

The system of claim 7, wherein the at least one processor is configured with instructions to translate the audio from the first language to the second language.

### Claim 10 (depends on 7)

The system of claim 7, wherein the at least one processor is configured with instructions to access the audio in the second language from an advanced television systems committee (ATSC) data stream.

### Claim 11 (depends on 7)

The system of claim 7, wherein the at least one processor comprises at least a first processor in the first recipient device configured to access the audio in second language and at least a second processor in a second recipient device configured to access the audio in a third language.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first and second processors are configured with instructions to translate the audio from the first language to a different language.