# US 2021/0174003 A1 · Sentence encoding and decoding method, storage medium, and device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174003 A1
- Document ID: US20210174003A1
- Status: Published application
- Kind: A1
- Application number: 17/181,490
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2018-11-29
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Fandong MENG; Jinchao ZHANG; Jie ZHOU
- Web page: https://patentgrep.com/patent/US20210174003A1

## CPC

- G06F 40/126: Character encoding
- G06F 40/151: Transformation
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of this disclosure disclose a sentence processing method and device. The method may include performing word segmentation operation on a source sentence to be encoded to obtain m words. The method may further include obtaining an ith word in the m words using an ith encoding processing node in the n encoding processing nodes, and obtaining an (i−1)th word vector from an (i−1)th encoding processing node. The method may further include performing linear operation and non-linear operation on the ith word and the (i−1)th word vector using the first unit of the ith encoding processing node to obtain an ith operation result, and outputting the ith operation result to the at least one second unit for processing to obtain an ith word vector. The method may further include generating, in response to obtaining m word vectors, a sentence vector according to the m word vectors.

## Claims

### Claim 1

A method for processing a sentence, performed by a sentence processing device and applicable to an encoding model, the encoding model comprising n cascaded encoding processing nodes, the encoding processing nodes comprising a first unit and at least one second unit that are cascaded, n>2, the method comprising:
performing word segmentation operation on a source sentence to be encoded to obtain m words, m<n;
obtaining an ith word in the m words using an ith encoding processing node in the n encoding processing nodes;
obtaining an (i−1)th word vector from an (i−1)th encoding processing node, the (i−1)th word vector being an encoding vector of an (i−1)th word in the m words, i<m;
performing linear operation and non-linear operation on the ith word and the (i−1)th word vector using the first unit of the ith encoding processing node to obtain an ith operation result;
outputting the ith operation result to the at least one second unit for processing to obtain an ith word vector; and
generating, in response to obtaining m word vectors, a sentence vector according to the m word vectors, the sentence vector being for determining a target sentence or a target category.

### Claim 2 (depends on 1)

The method of claim 1, wherein in response to the encoding model being a unidirectional encoding model and an encoding direction being from front to back,
the ith encoding processing node is a encoding processing node arranged at an ith position in the n encoding processing nodes in front-to-back order; and
the ith word is a word arranged at an ith position in the m words in front-to-back order.

### Claim 3 (depends on 1)

The method of claim 1, wherein in response to the encoding model being a unidirectional encoding model and an encoding direction being from back to front,
the ith encoding processing node is a encoding processing node arranged at an ith position in the n encoding processing nodes in back-to-front order; and
the ith word is a word arranged at an ith position in the m words in back-to-front order.

### Claim 4 (depends on 1)

The method of claim 1, wherein in response to the encoding model being a bidirectional encoding model and encoding directions are from front to back and from back to front, and m<n/2,
the ith encoding processing node comprises a encoding processing node arranged at an ith position in the n encoding processing nodes in front-to-back order and a encoding processing node arranged at an ith position in the n encoding processing nodes in back-to-front order; and
the ith word comprises a word arranged at an ith position in the m words in front-to-back order and a word arranged at an ith position in the m words in back-to-front order.

### Claim 5 (depends on 1)

The method of one claim 1, wherein performing linear operation and non-linear operation on the ith word and the (i−1)th word vector using the first unit of the ith encoding processing node comprises:
performing element-wise multiplication of the (i−1)th word vector and a first difference using the first unit to obtain a first product, wherein the first difference is equal to a value obtained by subtracting an update gate of the first unit from a predetermined value, and the update gate is for measuring a ratio of a portion from the ith word and a portion from the (i−1) word vector in the ith word vector;
performing linear transformation on the ith word using the first unit through a linear transformation function to obtain a linear transformation function value;
performing element-wise multiplication of the linear transformation function value and a linear transformation gate to obtain a second product;
performing non-linear transformation on the ith word and the (i−1)th word vector through a hyperbolic tangent function to obtain a hyperbolic tangent function value;
adding the hyperbolic tangent function value and the second product to obtain a candidate activation function value, wherein the linear transformation gate is for controlling the candidate activation function value to comprise the linear transformation function value;
performing element-wise multiplication of the update gate and the candidate activation function value using the first unit to obtain a third product; and
adding the first product and the third product using the first unit to obtain the ith operation result.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first unit is a gate recurrent unit having a non-linear operation capability and a linear operation capability, and the second unit is a transition gate recurrent unit.

### Claim 7 (depends on 1)

The method of claim 1, further comprising a decoding operation using a decoding model, wherein the decoding model comprises a decoding processing node, the decoding processing node comprises a third unit and at least one fourth unit that are cascaded, and the decoding operation comprises:
obtaining the sentence vector and a jth query state at a jth moment, the jth query state being for querying an encoded portion in the source sentence at the jth moment;
generating a jth source-language-focused context according to the sentence vector and the jth query state, the jth source-language-focused context being the encoded portion in the source sentence at the jth moment;
performing linear operation and non-linear operation on the jth query state and the jth source-language-focused context using the third unit of the decoding processing node to obtain an jth operation result; and
outputting the jth operation result to the at least one fourth unit of the decoding processing node for processing to obtain a jth word; and
generating, in response to obtaining k words, a target sentence according to the k words, j≤k.

### Claim 8

A method for decoding a sentence, performed by a sentence processing device and applicable to a decoding model, the decoding model comprising a decoding processing node, the decoding processing node comprising a first unit and at least one second unit that are cascaded, the method comprising:
obtaining a sentence vector and a jth query state at a jth moment, the sentence vector being obtained by encoding a source sentence using an encoding model, the jth query state being for querying an encoded portion in the source sentence at the jth moment;
generating a jth source-language-focused context according to the sentence vector and the jth query state, the jth source-language-focused context being the encoded portion in the source sentence at the jth moment;
performing linear operation and non-linear operation on the jth query state and the jth source-language-focused context using the first unit of the decoding processing node to obtain a jth operation result;
outputting the jth operation result to the at least one second unit of the decoding processing node for processing to obtain a jth word; and
generating, in response to obtaining k words, a target sentence according to the k words, j≤k.

### Claim 9 (depends on 8)

The method of claim 8, wherein the decoding model further comprises a query node communicatively connected to the decoding processing node, the query node comprises a linear transformation enhanced gate recurrent unit (L-GRU), and obtaining the jth query state comprises:
obtaining a (j−1)th decoded state and a (j−1)th word using the L-GRU of the query node, wherein the (j−1)th decoded state is obtained according to a (j−1)th operation result, and the (j−1)th decoded state is for determining the (j−1)th word; and
performing linear operation and non-linear operation on the (j−1)th decoded state and the (j−1)th word using the L-GRU of the query node to obtain the jth query state.

### Claim 10 (depends on 8)

The method of claim 8, wherein the decoding model further comprises a query node communicatively connected to the decoding processing node, the query node comprises a linear transformation enhanced gate recurrent unit (L-GRU) and at least one transition gate recurrent unit (T-GRU), and obtaining the jth query state comprises:
obtaining a (j−1)th decoded state and a (j−1)th word using the L-GRU of the query node, wherein the (j−1)th decoded state is obtained according to a (j−1)th operation result, and the (j−1)th decoded state is for determining the (j−1)th word;
performing linear operation and non-linear operation on the (j−1)th decoded state and the (j−1)th word using the L-GRU of the query node to obtain an intermediate operation result; and
outputting the intermediate operation result to the at least one T-GRU of the query node for processing to obtain the jth query state.

### Claim 11 (depends on 8)

The method of claim 8, wherein the decoding model further comprises a query node communicatively connected to the decoding processing node, the query node comprises a gate recurrent unit (GRU) and at least one transition gate recurrent unit (T-GRU); and obtaining the jth query state comprises:
obtaining a (j−1)th decoded state and a (j−1)th word using the GRU of the query node, wherein the (j−1)th decoded state is obtained according to a (j−1)th operation result, and the (j−1)th decoded state is for determining the (j−1)th word;
performing non-linear operation on the (j−1)th decoded state and the (j−1)th word using the GRU of the query node to obtain an intermediate operation result; and
outputting the intermediate operation result to the at least one T-GRU of the query node for processing to obtain the jth query state.

### Claim 12 (depends on 11)

The method of claim 11, wherein the decoding model further comprises an attention operation node, and the attention operation node is communicatively connected to the encoding model, the query node, and the decoding processing node, and generating the jth source-language-focused context according to the sentence vector and the jth query state comprises:
performing attention operation on the sentence vector and the jth query state using the attention operation node to obtain the jth source-language-focused context.

### Claim 13 (depends on 8)

The method of claim 8, wherein performing linear operation and non-linear operation on the jth query state and the jth source-language-focused context using the first unit of the decoding processing node comprises:
performing element-wise multiplication on the jth query state and a first difference using the first unit to obtain a first product, wherein the first difference is equal to a value obtained by subtracting an update gate of the first unit from a predetermined value, and the update gate is for measuring a ratio of a portion from the jth source-language-focused context and a portion from the jth query state in a jth source-language-focused context vector;
performing linear transformation on the jth source-language-focused context using the first unit through a linear transformation function to obtain a linear transformation function value;
performing element-wise multiplication on the linear transformation function value and a linear transformation gate to obtain a second product;
performing non-linear transformation on the jth source-language-focused context and the jth query state through a hyperbolic tangent function to obtain a hyperbolic tangent function value;
adding the hyperbolic tangent function value and the second product to obtain a candidate activation function value, wherein the linear transformation gate is for controlling the candidate activation function value to comprise the linear transformation function value;
performing element-wise multiplication on the update gate and the candidate activation function value using the first unit to obtain a third product; and
adding the first product and the third product using the first unit to obtain the jth operation result.

### Claim 14 (depends on 8)

The method of claim 8, wherein the first unit is a gate recurrent unit having a non-linear operation capability and a linear operation capability, and the second unit is a transition gate recurrent unit.

### Claim 15

A device for processing a sentence, comprising a memory and a processor, the memory storing computer-readable instructions and an encoding model, the encoding model comprising n cascaded encoding processing nodes, the encoding processing nodes comprising a first unit and at least one second unit that are cascaded, n>2, the computer-readable instructions, when executed by the processor, causing the processor to:
perform word segmentation operation on a source sentence to be encoded to obtain m words, m<n;
obtain an ith word in the m words using an ith encoding processing node in the n encoding processing nodes;
obtain an (i−1)th word vector from an (i−1)th encoding processing node, the (i−1)th word vector being an encoding vector of an (i−1)th word in the m words, i<m;
perform linear operation and non-linear operation on the ith word and the (i−1)th word vector using the first unit of the ith encoding processing node to obtain an ith operation result;
output the ith operation result to the at least one second unit for processing to obtain an ith word vector; and
generate, in response to obtaining m word vectors, a sentence vector according to the m word vectors, the sentence vector being for determining a target sentence or a target category.

### Claim 16 (depends on 15)

The device of claim 15, wherein in response to the encoding model being a unidirectional encoding model and an encoding direction being from front to back,
the ith encoding processing node is a encoding processing node arranged at an ith position in the n encoding processing nodes in front-to-back order; and
the ith word is a word arranged at an ith position in the m words in front-to-back order.

### Claim 17 (depends on 15)

The device of claim 15, wherein in response to the encoding model being a unidirectional encoding model and an encoding direction being from back to front,
the ith encoding processing node is a encoding processing node arranged at an ith position in the n encoding processing nodes in back-to-front order; and
the ith word is a word arranged at an ith position in the m words in back-to-front order.

### Claim 18 (depends on 15)

The device of claim 15, wherein in response to the encoding model being a bidirectional encoding model and encoding directions are from front to back and from back to front, and m<n/2,
the ith encoding processing node comprises a encoding processing node arranged at an ith position in the n encoding processing nodes in front-to-back order and a encoding processing node arranged at an ith position in the n encoding processing nodes in back-to-front order; and
the ith word comprises a word arranged at an ith position in the m words in front-to-back order and a word arranged at an ith position in the m words in back-to-front order.

### Claim 19 (depends on 15)

The device of claim 15, wherein the computer-readable instructions, when executed by the processor, cause the processor to:
perform element-wise multiplication of the (i−1)th word vector and a first difference using the first unit to obtain a first product, wherein the first difference is equal to a value obtained by subtracting an update gate of the first unit from a predetermined value, and the update gate is for measuring a ratio of a portion from the ith word and a portion from the (i−1)th word vector in the ith word vector;
perform linear transformation on the ith word using the first unit through a linear transformation function to obtain a linear transformation function value;
perform element-wise multiplication of the linear transformation function value and a linear transformation gate to obtain a second product;
perform non-linear transformation on the ith word and the (i−1)th word vector through a hyperbolic tangent function to obtain a hyperbolic tangent function value;
add the hyperbolic tangent function value and the second product to obtain a candidate activation function value, wherein the linear transformation gate is for controlling the candidate activation function value to comprise the linear transformation function value;
perform element-wise multiplication of the update gate and the candidate activation function value using the first unit to obtain a third product; and
add the first product and the third product using the first unit to obtain the ith operation result.

### Claim 20

A non-transitory machine-readable storage medium, having processor executable instructions and an encoding model stored thereon, the encoding model comprising n cascaded encoding processing nodes, the encoding processing nodes comprising a first unit and at least one second unit that are cascaded, n>2, the processor executable instructions being for causing a processor to:
perform word segmentation operation on a source sentence to be encoded to obtain m words, m<n;
obtain an ith word in the m words using an ith encoding processing node in the n encoding processing nodes;
obtain an (i−1)th word vector from an (i−1)th encoding processing node, the (i−1)th word vector being an encoding vector of an (i−1)th word in the m words, i<m;
perform linear operation and non-linear operation on the ith word and the (i−1)th word vector using the first unit of the ith encoding processing node to obtain an ith operation result;
output the ith operation result to the at least one second unit for processing to obtain an ith word vector; and
generate, in response to obtaining m word vectors, a sentence vector according to the m word vectors, the sentence vector being for determining a target sentence or a target category.
