# US 12,602,553 B2 · Speech translation method, device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,553 B2
- Document ID: US12602553B2
- Status: Granted
- Kind: B2
- Application number: 18/245,802
- Filed: 2021-09-02
- Published: 2024-01-25
- Granted: 2026-04-14
- Earliest claimed: 2020-09-18
- Assignees: Beijing Bytedance Network Technology Co., Ltd.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.
- Inventors: Lei Li; Mingxuan Wang; Qianqian Dong; Chengqi Zhao
- Web page: https://patentgrep.com/patent/US12602553B2

## CPC

- G06F 40/47: Machine-assisted translation, e.g. using translation memory
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F 40/30: Semantic analysis
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Provided are a speech translation method, a device, and a storage medium. The method includes: extracting, through an encoder of an end-to-end speech translation model, the semantic feature of a to-be-processed speech; decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature; decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature; and splitting the text sequence to obtain a target language text corresponding to the to-be-processed speech.

## Claims

### Claim 1

A speech translation method, comprising:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the encoder comprises a first encoder and a second encoder, and extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech comprises:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.

### Claim 3 (depends on 2)

The method according to claim 2, wherein detecting the blank frame and the duplicate frame in the acoustic feature comprises:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.

### Claim 4

An electronic device, comprising a processor and a memory storing computer programs, wherein the computer programs, when executed by the processor, implement the following:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.

### Claim 5

A non-transitory computer-readable storage medium storing computer programs, wherein the computer programs, when executed by a processor, implement the following:
extracting, through an encoder of an end-to-end speech translation model, a semantic feature of a to-be-processed speech;
decoding, through a decoder of the end-to-end speech translation model, a source language text corresponding to the semantic feature from the semantic feature;
decoding, through the decoder of the end-to-end speech translation model, the semantic feature according to the source language text to obtain a text sequence corresponding to the semantic feature, wherein the text sequence comprises the source language text and a target language text corresponding to the source language text; and
splitting the text sequence to obtain the target language text corresponding to the to-be-processed speech,
wherein a training process of the end-to-end speech translation model comprises:
pre-training, according to a text translation sample, the decoder of the end-to-end speech translation model to obtain an initial decoder; and
initializing the decoder of the end-to-end speech translation model based on the initial decoder; and training the initialized end-to-end speech translation model,
wherein the text translation sample comprises a source language sample text and a target language sample text corresponding to the source language sample text, and
wherein pre-training, according to the text translation sample, the decoder of the end-to-end speech translation model to obtain the initial decoder comprises:
splicing the source language sample text and the target language sample text to obtain a spliced sample sequence; and
pre-training, based on a masked cross-entropy loss function, the decoder by using the source language sample text and an all-zero vector as input of the decoder of the end-to-end speech translation model and the spliced sample sequence as desired output to obtain the initial decoder, wherein the masked cross-entropy loss function is configured to mask a prediction loss of a source language prediction text corresponding to the all-zero vector.

### Claim 6 (depends on 2)

The method according to claim 2, wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.

### Claim 7 (depends on 4)

The electronic device according to claim 4, wherein the encoder comprises a first encoder and a second encoder, and the computer programs implement extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech by:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.

### Claim 8 (depends on 7)

The electronic device according to claim 7, wherein the computer programs implement detecting the blank frame and the duplicate frame in the acoustic feature by:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.

### Claim 9 (depends on 5)

The storage medium according to claim 5, wherein the encoder comprises a first encoder and a second encoder, and the computer programs implement extracting, through the encoder of the end-to-end speech translation model, the semantic feature of the to-be-processed speech by:
extracting, through the first encoder, an acoustic feature of the to-be-processed speech;
detecting a blank frame and a duplicate frame in the acoustic feature, eliminating the blank frame and combining the duplicate frame to obtain a contracted acoustic feature; and
extracting, through the second encoder, the semantic feature in the contracted acoustic feature.

### Claim 10 (depends on 4)

The electronic device according to claim 4, wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.

### Claim 11 (depends on 9)

The storage medium according to claim 9, wherein detecting the blank frame and the duplicate frame in the acoustic feature comprises:
detecting, based on a spike characteristic of a probability distribution of a connectionist temporal classification loss function, the blank frame and the duplicate frame in the acoustic feature.

### Claim 12 (depends on 5)

The storage medium according to claim 5, wherein a training process of the first encoder comprises:
training, based on a connectionist temporal classification loss function, the first encoder by using a sample speech in a training sample set of the end-to-end speech translation model as input of the first encoder and a sample phoneme sequence corresponding to the sample speech as desired output.