# US 2022/0092274 A1 · Re-translation for simultaneous, spoken-language machine translation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092274 A1
- Document ID: US20220092274A1
- Status: Published application
- Kind: A1
- Application number: 17/030,093
- Filed: 2020-09-23
- Published: 2022-03-24
- Earliest claimed: 2020-09-23
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Naveen Arivazhagan; Colin Andrew Cherry; Wolfgang Macherey; Te I; George Foster; Pallavi N. Baljekar
- Web page: https://patentgrep.com/patent/US20220092274A1

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 40/47: Machine-assisted translation, e.g. using translation memory
- G10L 15/16: using artificial neural networks
- G10L 15/19: Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for re-translation for simultaneous, spoken-language machine translation. In some implementations, a stream of audio data comprising speech in a first language is received. A transcription for the speech in the stream of audio data is generated using an automated speech recognizer through a series of updates. A translation of the transcription into a second language is generated using a machine translation module. The translation is generated with translation iterations that translate increasing amounts of the transcription, including re-translating previously portions of the transcription. A series of translation updates are provided to a client device based on the translation iterations.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
receiving, by the one or more computers, a stream of audio data comprising speech in a first language;
generating, by the one or more computers, a transcription for the speech in the stream of audio data using an automated speech recognizer, the transcription being incrementally generated by the automated speech recognizer through a series of updates that add additional text in the first language to the transcription;
generating, by the one or more computers, a translation of the transcription into a second language using a machine translation module, the translation being generated by performing, in response to each of the updates that add additional text in the first language to the transcription, a corresponding translation iteration that translates into the second language an end portion of the transcription, wherein the end portion for at least some of the translation iterations comprises (i) the additional text corresponding to the update and (ii) previously-translated text of the transcription, wherein, for at least some of the translation iterations, the one or more computers promote stability of the translation by biasing output toward including the previously-translated text of the transcription; and
providing, by the one or more computers and over a communication network, a series of translation updates to a client device based on the translation iterations, wherein the translation updates respectively provide portions of the translated text from the corresponding translation iterations.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the transcription, generating the translation, and providing the translation updates are performed concurrently with receiving additional audio data in the stream of audio data.

### Claim 3 (depends on 1)

The method of claim 1, wherein each of the end portions includes a last full sentence of the transcription or a last partial sentence in the transcription.

### Claim 4 (depends on 1)

The method of claim 1, wherein performing the translation iterations comprises:
performing a first translation iteration that translates a first word in a sentence; and
performing successive translation iterations, as additional words in the sentence are received from the automated speech recognizer, that each retranslate the first word in the sentence along with subsequently received text of the sentence, such that the text translated in the successive iterations is expanded at each successive iteration until a translation iteration for the full sentence is performed.

### Claim 5 (depends on 1)

The method of claim 1, wherein providing the series of translation updates comprises omitting, from each of the translation updates that do not correspond to the translation of a full sentence, a predetermined number of tokens at the end of the sentence.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the translation comprises biasing the translation iterations toward consistency with translation results of one or more prior translation iterations.

### Claim 7 (depends on 6)

The method of claim 6, wherein biasing the translation comprises biasing a beam search process such that, for a second or subsequent translation iteration for a sentence, the biasing increases a likelihood of selection of a beam that includes the results of the immediately previous translation iteration for the sentence.

### Claim 8 (depends on 7)

The method of claim 7, wherein biasing the beam search process comprises:
providing (i) a sequence of one or more source tokens to be translated, and (ii) a sequence of one or more output tokens from the previous translation iteration; and
determining a score for an output token in the beam search by interpolating between (i) a posterior distribution provided by the machine translation module and (ii) a distribution representing the sequence of one or more output tokens from the previous translation iteration.

### Claim 9 (depends on 6)

The method of claim 6, wherein biasing the translation comprises biasing toward a beam that has strictly followed a target sequence representing a previous translation for sentence currently being translated; and
in response to detecting that a beam diverges from the target sequence, removing the bias for the beam.

### Claim 10 (depends on 1)

The method of claim 1, wherein the machine translation module comprises a neural machine translation model.

### Claim 11 (depends on 1)

The method of claim 1, comprising applying one or more post-processing operations to the output of the automated speech recognizer to increase stability of transcription results before providing transcribed text to the machine translation module.

### Claim 12 (depends on 11)

The method of claim 11, wherein the one or more post-processing operations comprise:
evaluating, for individual tokens of transcribed text, a stability score indicative of a probability that the token will change in a future update by the automated speech recognizer; and
suppressing tokens for which the stability score does not satisfy a predetermined threshold from being provided to the machine translation module for translation.

### Claim 13 (depends on 11)

The method of claim 11, wherein the one or more post-processing operations comprise omitting, from the text provided to the machine translation module, punctuation that denotes the end of a sentence until a predetermined condition is determined.

### Claim 14 (depends on 13)

The method of claim 13, wherein the predetermined condition comprises detection of a speech utterance endpoint.

### Claim 15 (depends on 11)

The method of claim 11, wherein the one or more post-processing operations comprise omitting, from the text provided to the machine translation module, a predetermined number of tokens from the end of the automated speech recognizer output.

### Claim 16 (depends on 11)

The method of claim 11, wherein the one or more post-processing operations comprise limiting updates to the transcription for the text stream to appending additional words without revising existing words in the transcription.

### Claim 17 (depends on 16)

The method of claim 16, comprising, in response to determining that a transcription for a portion of audio data differs from a prior transcription for the portion audio data:
identifying a shortest prefix portion of a current transcription of the sentence that can be replaced with the prior transcription to minimize an edit distance between the current transcription and the prior transcription the sentence;
generating an altered version of the current transcription that replaces the identified shortest prefix portion with the prior transcription of the sentence; and
providing the generated altered version of the current transcription to the machine translation module for translation into the second language.

### Claim 18 (depends on 1)

The method of claim 1, wherein the edit distance is a Levenshtein distance over characters.

### Claim 19

A system comprising:
one or more computers; and
one or more computer-readable media storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform operations comprising:
receiving, by the one or more computers, a stream of audio data comprising speech in a first language;
generating, by the one or more computers, a transcription for the speech in the stream of audio data using an automated speech recognizer, the transcription being incrementally generated by the automated speech recognizer through a series of updates that add additional text in the first language to the transcription;
generating, by the one or more computers, a translation of the transcription into a second language using a machine translation module, the translation being generated by performing, in response to each of the updates that add additional text in the first language to the transcription, a corresponding translation iteration that translates into the second language an end portion of the transcription, wherein the end portion at least some of the translation iterations comprises (i) the additional text corresponding to the update and (ii) previously-translated text of the transcription, wherein, for at least some of the translation iterations, the one or more computers promote stability of the translation by biasing output toward including the previously-translated text of the transcription; and
providing, by the one or more computers and over a communication network, a series of translation updates to a client device based on the translation iterations, wherein the translation updates respectively provide portions of the translated text from the corresponding translation iterations.

### Claim 20

One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, are operable to cause the one or more computers to perform operations comprising:
receiving, by the one or more computers, a stream of audio data comprising speech in a first language;
generating, by the one or more computers, a transcription for the speech in the stream of audio data using an automated speech recognizer, the transcription being incrementally generated by the automated speech recognizer through a series of updates that add additional text in the first language to the transcription;
generating, by the one or more computers, a translation of the transcription into a second language using a machine translation module, the translation being generated by performing, in response to each of the updates that add additional text in the first language to the transcription, a corresponding translation iteration that translates into the second language an end portion of the transcription, wherein the end portion at least some of the translation iterations comprises (i) the additional text corresponding to the update and (ii) previously-translated text of the transcription, wherein, for at least some of the translation iterations, the one or more computers promote stability of the translation by biasing output toward including the previously-translated text of the transcription; and
providing, by the one or more computers and over a communication network, a series of translation updates to a client device based on the translation iterations, wherein the translation updates respectively provide portions of the translated text from the corresponding translation iterations.
