# US 2019/0172466 A1 · Language processing method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172466 A1
- Document ID: US20190172466A1
- Status: Published application
- Kind: A1
- Application number: 16/108,717
- Filed: 2018-08-22
- Published: 2019-06-06
- Earliest claimed: 2017-12-04
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Min-Joong LEE; Hodong LEE
- Web page: https://patentgrep.com/patent/US20190172466A1

## CPC

- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06F 40/216: using statistical methods
- G06F 40/242: Dictionaries
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/44: Statistical methods, e.g. probability models
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A language processing method and apparatus is disclosed. A language processing apparatus using a neural network may obtain context information from a source text using a neural network-based encoder, generate a prefix token from the context information using a neural network-based main decoder, generate a token sequence including at least two successive tokens sequentially following the prefix token using a skip model in response to the prefix token satisfying a preset condition, and indicate a target text in which the prefix token and the token sequence are combined as an inference result with respect to the source text.

## Claims

### Claim 1

A processor implemented method for language processing using a neural network, the method comprising:
obtaining context information from a source text using a neural network-based encoder;
generating a prefix token from the context information using a neural network-based main decoder;
in response to the prefix token satisfying a preset condition, generating a token sequence including at least two successive tokens sequentially following the prefix token using a skip model; and
indicating a target text in which the prefix token and the token sequence are combined as an inference result with respect to the source text.

### Claim 2 (depends on 1)

The method of claim 1, further comprising inputting the prefix token and the token sequence to the main decoder to sequentially generate one or more tokens sequentially following the token sequence.

### Claim 3 (depends on 1)

The method of claim 1, wherein the main decoder is configured to generate tokens sequentially.

### Claim 4 (depends on 3)

The method of claim 3, wherein, based on the prefix token satisfying the preset condition and through the generating of the token sequence by the skip model, the main decoder non-sequentially decodes the context information.

### Claim 5 (depends on 4)

The method of claim 4, wherein the main decoder generates tokens sequentially, except that when the skip model generates a corresponding token sequence for a corresponding prefix token the main decoder skips decoding of tokens sequentially following the corresponding prefix token by next inputting, sequentially after the generating of the corresponding prefix token, the corresponding prefix token and the corresponding token sequence to the main decoder for decoding processing by the main decoder.

### Claim 6 (depends on 3)

The method of claim 3, wherein the main decoder is a non-recurrent neural network.

### Claim 7 (depends on 1)

The method of claim 1,
wherein the encoder is trained to perform a vector transformation of the source text in a first language for the obtaining of the context information, and
wherein the main decoder is trained to perform a translation of the context information into probabilistic information with respect to a different second language corresponding to the target text.

### Claim 8 (depends on 1)

The method of claim 1, in response to the prefix token not satisfying the condition, further comprising:
generating a token sequentially following the prefix token using the main decoder.

### Claim 9 (depends on 1)

The method of claim 1, wherein the skip model includes a dictionary in which a plurality of words are arranged in alphabetical order.

### Claim 10 (depends on 9)

The method of claim 9, wherein the condition includes a condition that a probability of the token sequence following the prefix token is greater than or equal to a threshold value.

### Claim 11 (depends on 1)

The method of claim 1, wherein the skip model includes a neural network-based sub-decoder.

### Claim 12 (depends on 11)

The method of claim 11, wherein the condition includes a condition that a degree of difficulty of generating the token sequence by the skip model is less than or equal to a threshold value.

### Claim 13 (depends on 12)

The method of claim 12, wherein the sub-decoder includes first hidden nodes and the main decoder includes second hidden nodes,
wherein one or more hidden nodes of the second hidden nodes generates a token of the prefix token,
at least one hidden node of the first hidden nodes is configured to receive the prefix token and one or more hidden nodes of the first hidden nodes generate the token sequence, and
based on whether the prefix token satisfies the preset condition, another hidden node of the second hidden nodes is configured to receive the prefix token and the token sequence.

### Claim 14 (depends on 1)

The method of claim 1,
wherein the generating of the prefix token comprises generating a prefix token for each of a preset number of beam nodes, and
wherein the generating of the token sequence comprises generating respective token sequences using the skip model for each of the beam nodes.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
determining the target text from among candidate texts respectively including the prefix token and the generated respective token sequences for each of the beam nodes.

### Claim 16 (depends on 1)

The method of claim 1, wherein the skip model includes first hidden nodes and the main decoder includes second hidden nodes,
one or more hidden nodes of the second hidden nodes generate a token of the prefix token,
at least one hidden node of the first hidden nodes is configured to receive the prefix token and one or more hidden nodes of the first hidden nodes generate the token sequence, and
based on whether the prefix token satisfies the preset condition, another hidden node of the second hidden nodes is configured to receive the prefix token and the token sequence.

### Claim 17 (depends on 1)

The method of claim 1, wherein the token includes a character, plural characters, a word, or a sentence.

### Claim 18 (depends on 1)

The method of claim 1, wherein the target text corresponds to a translation result of the source text.

### Claim 19 (depends on 1)

The method of claim 1, wherein the target text is a result of generating a natural language corresponding to the source text.

### Claim 20 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

### Claim 21

A language processing apparatus comprising:
one or more processors configured to:
generate context information from a source text using a neural network-based encoder;
generate a prefix token from the context information using a neural network-based main decoder;
in response to the prefix token satisfying a preset condition, generate the token sequence sequentially following the prefix token using a skip model configured to generate a token sequence including at least two successive tokens; and
indicate a target text in which the prefix token and the token sequence are combined as an inference result with respect to the source text.

### Claim 22 (depends on 21)

The language processing apparatus of claim 21, wherein the one or more processors are further configured to input the prefix token and the token sequence to the main decoder to sequentially generate one or more tokens sequentially following the token sequence.

### Claim 23 (depends on 21)

The language processing apparatus of claim 21, wherein the main decoder is configured to generate tokens sequentially.

### Claim 24 (depends on 23)

The language processing apparatus of claim 23, wherein, based on the prefix token satisfying the preset condition and through the generating of the token sequence by the skip model, the main decoder non-sequentially decodes the context information.

### Claim 25 (depends on 24)

The language processing apparatus of claim 24, wherein, when the skip model generates a corresponding token sequence for a corresponding prefix token, the main decoder skips decoding of tokens sequentially following the corresponding prefix token by next inputting, sequentially after the generating of the corresponding prefix token, the corresponding prefix token and the corresponding token sequence to the main decoder for further decoding processing by the main decoder.

### Claim 26 (depends on 21)

The language processing apparatus of claim 21,
wherein the encoder is trained to perform a vector transformation of the source text in a first language for the obtaining of the context information, and
wherein the main decoder is trained to perform a translation of the context information into probabilistic information with respect to a different second language corresponding to the target text.

### Claim 27 (depends on 21)

The language processing apparatus of claim 21, wherein, in response to the prefix token not satisfying the condition, the one or more processors are further configured to generate a token sequentially following the prefix token using the main decoder.

### Claim 28 (depends on 21)

The language processing apparatus of claim 21, wherein the skip model includes a dictionary in which a plurality of words are arranged in alphabetical order.

### Claim 29 (depends on 21)

The language processing apparatus of claim 21, wherein the skip model includes a neural network-based sub-decoder.

### Claim 30 (depends on 21)

The language processing apparatus of claim 21, further comprising a memory comprising instructions, which when executed by the one or more processors, cause the one or more processors to be configured to implement the generating of the context information, generating of the prefix token, indicating of the target text, and the generating of the token sequence in response to the one or more processors determining that the prefix token satisfies the preset condition.

### Claim 31

A processor implemented method for language processing using a neural network, the language processing method comprising:
decoding tokens from context information for a source text using a neural network-based main decoder, including sequentially decoding one or more tokens to identify a prefix token;
sequentially following the decoding of the one or more tokens, selectively controlling the decoding of the tokens to be one of performed sequentially, to generate a next token sequentially following the decoding of the one or more tokens, and performed non-sequentially in bulk, to skip decoding of plural next tokens sequentially following the decoding of the one or more tokens; and
identifying a target text based on a combination of the prefix token with a select one of the next token and the plural next tokens dependent on the selective control of the decoding of the tokens.

### Claim 32 (depends on 31)

The method of claim 31, further comprising indicating the target text as an inference result with respect to the source text.

### Claim 33 (depends on 31)

The method of claim 31, further comprising obtaining the context information from the source text using a neural network-based encoder.

### Claim 34 (depends on 33)

The method of claim 33,
wherein the encoder is trained to perform a vector transformation of the source text in a first language for the obtaining of the context information, and
wherein the main decoder is trained to perform a translation of the context information into probabilistic information with respect to a different second language corresponding to the target text.

### Claim 35 (depends on 31)

The method of claim 31, wherein the selective controlling of the decoding of the tokens is dependent on a determination of whether the prefix token satisfies a preset condition.

### Claim 36 (depends on 35)

The method of claim 35, further comprising, based on the determination, generating the plural next tokens using, as a skip model, a neural network-based sub-decoder.

### Claim 37 (depends on 36)

The method of claim 36, wherein the determination includes determining whether a degree of difficulty of generating the plural next tokens is less than or equal to a threshold value.

### Claim 38 (depends on 35)

The method of claim 35, further comprising, based on the determination, generating the plural next tokens using a skip model dictionary including plural words arranged in an alphabetical order and relatedness information with respect to differences in similarities respectively between the plural words.

### Claim 39 (depends on 38)

The method of claim 38, wherein the determination includes determining whether a probability of the plural next tokens following the prefix token is greater than or equal to a threshold value.

### Claim 40 (depends on 31)

The method of claim 31,
wherein the decoding of the tokens includes identifying a prefix token for each of a preset number of beam nodes, and
wherein the controlling of the decoding of tokens to be performed non-sequentially in bulk includes generating respective plural next tokens for each of the beam nodes.

### Claim 41 (depends on 40)

The method of claim 40, wherein the identifying of the target text is further based on a determination of the target text from among candidate texts respectively including the prefix token and the generated respective plural next tokens for each of the beam nodes.
