# US 10,372,831 B2 · Auto-translation for multi user audio and video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,831 B2
- Document ID: US10372831B2
- Status: Granted
- Kind: B2
- Application number: 15/664,706
- Filed: 2017-07-31
- Published: 2017-12-14
- Granted: 2019-08-06
- Earliest claimed: 2011-12-12
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Trausti Kristjansson; John Huang; Yu-Kuan Lin; Hung-ying Tyan; Jakob David Uszkoreit; Joshua James Estelle; Chung-yih Wang; Kirill Buryak; Yusuke Konishi
- Web page: https://patentgrep.com/patent/US10372831B2

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 3/56: Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities (video conference systems H04N7/15)
- H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)}
- H04N 7/15: Conference systems
- H04N 7/152: {Multipoint control units therefor}
- H04M 2203/2061: Language aspects
- H04M 2242/12: Language recognition, selection or translation arrangements
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The disclosed subject matter provides a system, computer readable storage medium, and a method providing an audio and textual transcript of a communication. A conferencing services may receive audio or audio visual signals from a plurality of different devices that receive voice communications from participants in a communication, such as a chat or teleconference. The audio signals representing voice (speech) communications input into respective different devices by the participants. A translation services server may receive over a separate communication channel the audio signals for translation into a second language. As managed by the translation services server, the audio signals may be converted into textual data. The textual data may be translated into text of different languages based the language preferences of the end user devices in the teleconference. The translated text may be further translated into audio signals.

## Claims

### Claim 1

A system comprising:
a communication server configured to communicate with two or more end user devices, each including an audio input device and an audio output device, and the communication server is further configured to perform operations comprising:
receiving audio data signals representing communication in a first spoken language from a first end user device; and
transmitting the audio data signals to a second end user device via a first communication channel, the audio data signals when received by the second end user device causing the second end user device to embed a language preference for the second end user device into the audio data signals; and
a translation services server configured to communicate with the end user devices and perform operations comprising:
after the second end user device embeds the language preference into the audio data signals, receiving the audio data signals from the second end user device via a second communication channel that is separate from the first communication channel between the second end user device and the communication server;
executing a language recognition process to recognize the first spoken language of the communication represented by the audio data signals;
determining whether the first spoken language of the communication represented by the audio data signals corresponds to the language preference of the second end user device;
in response to determining that the first spoken language does not correspond to the language preference of the second end user device, translating the audio data signals in the first spoken language into audio data signals in a second spoken language, the second spoken language corresponding to the language preference of the second end user device; and
transmitting the translated audio data signals to the second end user device.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a speech-to-text processor configured to convert the first spoken language audio data signals into first language text corresponding to the communication in the first spoken language.

### Claim 3 (depends on 2)

The system of claim 2, further comprising a translation processor configured to translate the first language text into second language text.

### Claim 4 (depends on 3)

The system of claim 3, further comprising a text-to-speech processor configured to convert the second language text into audio data signals representing a spoken version of the second language text.

### Claim 5

A method comprising:
receiving, at a translation services server, audio data signals from a first end user device via a first communication channel, the audio data signals representing communication in a first spoken language and having an embedding of a language preference for the first end user device, wherein the first end user device is configured to:
receive the audio data signals from a communication server via a second communication channel that is separate from the first communication channel between the first end user device and the translation services server, the communication server receiving the audio data signals representing the communication in the first spoken language from a second end user device; and
embed the language preference for the first end user device into the audio data signals received from the communication server;
executing, by the translation services server, a language recognition process to recognize the first spoken language of the communication represented by the audio data signals;
determining, by the translation services server, whether the first spoken language of the communication represented by the audio data signals corresponds to the language preference of the first end user device;
in response to determining that the first spoken language does not correspond to the language preference of the first end user device, translating, by the translation services server, the audio data signals in the first spoken language into audio data signals in a second spoken language, the second spoken language corresponding to the language preference of the first end user device; and
transmitting the translated audio data signals from the translation services server to the first end user device.

### Claim 6 (depends on 5)

The method of claim 5, further comprising sending the first spoken language audio data signals from the translation services server to a speech-to-text processor configured to convert the first spoken language audio data signals into first language text corresponding to the communication in the first spoken language.

### Claim 7 (depends on 6)

The method of claim 6, further comprising sending the first language text from the translation services server to a translation processor configured to translate the first language text into second language text.

### Claim 8 (depends on 7)

The method of claim 7, further comprising sending the second language text from the translation services server to a text-to-speech processor configured to convert the second language text into audio data signals representing a spoken version of the second language text.

### Claim 9

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising executing a communication service and a translation services;
wherein the communication service is configured to communicate with two or more end user devices each including an audio input device and an audio output device, and the communication service is further configured to perform operations comprising:
receiving audio data signals representing communication in a first spoken language from a first end user device; and
transmitting the audio data signals to a second end user device via a first communication channel, the audio data signals when received by the second end user device causing the second end user device to embed a language preference for the second end user device into the audio data signals; and
wherein the translation service is configured to communicate with the end user devices and perform operations comprising:
after the second end user device embeds the language preference into the audio data signals, receiving the audio data signals from the second end user device via a second communication channel that is separate from the first communication channel between the second end user device and the communication service;
executing a language recognition process to recognize the first spoken language of the communication represented by the audio data signals;
determining whether the first spoken language of the communication represented by the audio data signals corresponds to the language preference of the second end user device;
in response to determining that the first spoken language does not correspond to the language preference of the second end user device, translating the audio data signals in the first spoken language into audio data signals in a second spoken language, the second spoken language corresponding to the language preference of the second end user device; and
transmitting the translated audio data signals to the second end user device.

### Claim 10 (depends on 9)

The system of claim 9, further comprising a speech-to-text processor configured to convert the first spoken language audio data signals into first language text corresponding to the communication in the first spoken language.

### Claim 11 (depends on 10)

The system of claim 10, further comprising a translation processor configured to translate the first language text into second language text.

### Claim 12 (depends on 11)

The system of claim 11, further comprising a text-to-speech processor configured to convert the second language text into audio data signals representing a spoken version of the second language text.