# US 2021/0232775 A1 · Language generation method and apparatus, electronic device and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232775 A1
- Document ID: US20210232775A1
- Status: Published application
- Kind: A1
- Application number: 17/031,569
- Filed: 2020-09-24
- Published: 2021-07-29
- Earliest claimed: 2020-01-23
- Assignees: Beijing Baidu Netcom Science Technology Co., Ltd.
- Applicants: Beijing Baidu Netcom Science and Technology Co., Ltd.
- Inventors: Han ZHANG; Dongling XIAO; Yukun LI; Yu SUN; Hao TIAN; Hua WU; Haifeng WANG
- Web page: https://patentgrep.com/patent/US20210232775A1

## CPC

- G06F 40/56: Natural language generation
- G06F 40/30: Semantic analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure proposes a language generation method and apparatus. The method includes: performing encoding processing on an input sequence by using a preset encoder to generate a hidden state vector corresponding to the input sequence; in response to a granularity category of a second target segment being a phrase, decoding a first target segment vector, the hidden state vector, and a position vector corresponding to the second target segment by using N decoders to generate N second target segments; determining a loss value based on differences between respective N second target segments and a second target annotated segment; and performing parameter updating on the preset encoder, a preset classifier, and the N decoders based on the loss value to generate an updated language generation model for performing language generation.

## Claims

### Claim 1

A language generation method, comprising:
performing encoding processing on an input sequence by using a preset encoder to generate a hidden state vector corresponding to the input sequence;
classifying the hidden state vector and a generated first target segment vector by using a preset classifier to determine a granularity category of a second target segment to be predicted;
in response to the second target segment being a phrase, decoding the first target segment vector, the hidden state vector and a position vector corresponding to the second target segment by using N decoders to generate N second target segments, N being an integer greater than 1;
determining a loss value based on differences between respective N second target segments and a second target annotated segment; and
performing parameter updating on the preset encoder, the preset classifier, and the N decoders based on the loss value to generate an updated language generation model for performing language generation.

### Claim 2 (depends on 1)

The method of claim 1, wherein the second target segment comprises two characters, and the N decoders at least comprise a first decoder and a second decoder; and
decoding the first target segment vector, the hidden state vector and the position vector corresponding to the second target segment by using the N decoders to generate the N second target segments comprises:
decoding the first target segment vector, the hidden state vector, and a first position vector corresponding to the second target segment by using the first decoder to generate a first candidate character at a first position;
decoding the first target segment vector, a vector corresponding to the first candidate character, the hidden state vector, and a second position vector corresponding to the second target segment by using the first decoder to generate a first candidate character at a second position;
generating a first second target segment based on the first candidate character at the first position and the first candidate character at the second position;
decoding the first target segment vector, the hidden state vector and the first position vector by using the second decoder to generate a second candidate character at the first position;
decoding the first target segment vector, the hidden state vector and the second position vector by using the second decoder to generate a second candidate character at the second position; and
generating a second second target segment based on the second candidate character at the first position and the second candidate character at the second position.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first position vector corresponding to the second target segment is a position vector of a first target segment, and the second position vector corresponding to the second target segment comprises the position vector of the first target segment and a position vector of a first character in the second target segment.

### Claim 4 (depends on 2)

The method of claim 2, wherein the first position vector corresponding to the second target segment is a position vector of a first character in the second target segment, and the second position vector corresponding to the second target segment is a position vector of a second character in the second target segment,
before generating the N second target segments, the method further comprises:
obtaining a preset placeholder vector; and
generating the N second target segments comprises:
decoding the preset placeholder vector, the first target segment vector, the hidden state vector, and the position vector corresponding to the second target segment by using the N decoders to generate the N second target segments.

### Claim 5 (depends on 1)

The method of claim 1, after determining the granularity category of the second target segment to be predicted, further comprising:
in response to the second target segment being a character, decoding the first target segment vector, the hidden state vector and the position vector of the first target segment by using a decoder corresponding to the character to generate the second target segment.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the loss value based on the differences between respective N second target segments and the second target annotated segment comprises:
determining a loss reference value based on a difference between a second target segment and the second target annotated segment; and
determining the loss value based on loss reference values of the N second target segments and a preset weight.

### Claim 7 (depends on 6)

The method of claim 6, wherein performing the language generation by using the updated language generation model is executed by:
obtaining a supervised training sample based on a set language generation scene;
performing parameter adjustment on the updated language generation model based on the supervised training sample; and
performing the language generation in the set language generation scene by using the updated language generation model after the parameter adjustment.

### Claim 8 (depends on 7)

The method of claim 7, wherein the set language generation scene comprises one or a combination of dialogue generation, machine translation, question-answer scene and summarization generation.

### Claim 9

A language generation apparatus, comprising a non-transitory computer-readable medium including computer-executable instructions stored thereon, and an instruction execution system which is configured by the instructions to implement at least one of:
a first generation module, configured to perform encoding processing on an input sequence by using a preset encoder to generate a hidden state vector corresponding to the input sequence;
a first determination module, configured to classify the hidden state vector and a generated first target segment vector by using a preset classifier to determine a granularity category of a second target segment to be predicted;
a second generation module, configured to, in response to the second target segment being a phrase, decode the first target segment vector, the hidden state vector and a position vector corresponding to the second target segment by using N decoders to generate N second target segments, N being an integer greater than 1;
a second determination module, configured to determine a loss value based on differences between respective N second target segments and a second target annotated segment; and
a language generation module, configured to perform parameter updating on the preset encoder, the preset classifier, and the N decoders based on the loss value to generate an updated language generation model for performing language generation.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the second target segment comprises two characters, and the N decoders at least comprise a first decoder and a second decoder; and
the second generation module is configured to:
decode the first target segment vector, the hidden state vector, and a first position vector corresponding to the second target segment by using the first decoder to generate a first candidate character at a first position;
decode the first target segment vector, a vector corresponding to the first candidate character, the hidden state vector, and a second position vector corresponding to the second target segment by using the first decoder to generate a first candidate character at a second position;
generate a first second target segment based on the first candidate character at the first position and the first candidate character at the second position;
decode the first target segment vector, the hidden state vector and the first position vector by using the second decoder to generate a second candidate character at the first position;
decode the first target segment vector, the hidden state vector and the second position vector by using the second decoder to generate a second candidate character at the second position; and
generate a second second target segment based on the second candidate character at the first position and the second candidate character at the second position.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the first position vector corresponding to the second target segment is a position vector of a first target segment, and the second position vector corresponding to the second target segment comprises the position vector of the first target segment and a position vector of a first character in the second target segment.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the first position vector corresponding to the second target segment is a position vector of a first character in the second target segment, and the second position vector corresponding to the second target segment is a position vector of a second character in the second target segment,
the instruction execution system is further configured by the instructions to implement:
an obtaining module, configured to obtain a preset placeholder vector; and
the second generation module is further configured to:
decode the preset placeholder vector, the first target segment vector, the hidden state vector, and the position vector corresponding to the second target segment by using the N decoders to generate the N second target segments.

### Claim 13 (depends on 9)

The apparatus of claim 9, wherein the instruction execution system is further configured by the instructions to implement:
a third generation module, configured to, in response to the second target segment being a character, decode the first target segment vector, the hidden state vector and the position vector of the first target segment by using a decoder corresponding to the character to generate the second target segment.

### Claim 14 (depends on 9)

The apparatus of claim 9, wherein the second determination module is configured to:
determine a loss reference value based on a difference between a second target segment and the second target annotated segment; and
determine the loss value based on loss reference values of the N second target segments and a preset weight.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the language generation module is configured to:
obtain a supervised training sample based on a set language generation scene;
perform parameter adjustment on the updated language generation model based on the supervised training sample; and
perform the language generation in the set language generation scene by using the updated language generation model after the parameter adjustment.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the set language generation scene comprises one or a combination of dialogue generation, machine translation, question-answer scene and summarization generation.

### Claim 17 (depends on 1)

An electronic device, comprising:
at least one processor; and
a storage device communicatively connected to the at least one processor; wherein,
the storage device stores an instruction executable by the at least one processor, and when the instruction is executed by the at least one processor, the at least one processor may implement the method of claim 1.

### Claim 18

A non-transitory computer-readable storage medium having a computer instruction stored thereon, wherein the computer instruction is configured to make a computer implement a language generation method, and the method comprises:
performing encoding processing on an input sequence by using a preset encoder to generate a hidden state vector corresponding to the input sequence;
classifying the hidden state vector and a generated first target segment vector by using a preset classifier to determine a granularity category of a second target segment to be predicted;
in response to the second target segment being a phrase, decoding the first target segment vector, the hidden state vector and a position vector corresponding to the second target segment by using N decoders to generate N second target segments, N being an integer greater than 1;
determining a loss value based on differences between respective N second target segments and a second target annotated segment; and
performing parameter updating on the preset encoder, the preset classifier, and the N decoders based on the loss value to generate an updated language generation model for performing language generation.
