# US 2021/0342529 A1 · Word coding device, analysis device, language model learning device, method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342529 A1
- Document ID: US20210342529A1
- Status: Published application
- Kind: A1
- Application number: 17/269,665
- Filed: 2019-08-22
- Published: 2021-11-04
- Earliest claimed: 2018-08-24
- Assignees: Nippon Telegraph and Telephone Corporation
- Applicants: Nippon Telegraph and Telephone Corporation
- Inventors: Sho TAKASE; Masaaki NAGATA
- Web page: https://patentgrep.com/patent/US20210342529A1

## CPC

- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06F 40/30: Semantic analysis
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

To generate a vector representing a word in consideration of the importance of the character string of the word. A character N-gram vector acquisition unit acquires a vector representing each of character N-grams included in a word. A weight calculation unit 40 calculates a weight for a vector representing each of the character N-grams on the basis of the vector representing each of the character N-grams included in the word, and a character N-gram vector combining unit 42 generates a composite vector obtained by combining the vector representing each of the character N-grams on the basis of the vector representing each of the character N-grams included in the word and the weight for the vector representing each of the character N-grams.

## Claims

### Claim 1

A word coding device comprising a word coder configured to convert an entered word into a composite vector by using a pre-learned neural network, wherein
the entered word includes character N-grams,
the word coder includes:
a character N-gram vector acquirer configured to acquire a predetermined vector representing each of the character N-grams included in the entered word;
a weight determiner configured to calculate a weight for the predetermined vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the entered word; and
a character N-gram vector combiner configured to generate a composite vector obtained by combining the predetermined vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the entered word and the weight for the predetermined vector representing each of the character N-grams.

### Claim 2

An analysis device comprising an analyzer configured to output a prediction result of an output text for a predetermined purpose for an entered text by using a pre-learned neural network, wherein
the entered text includes words,
the analyzer includes:
a word coder configured to convert each word included in the entered text into a composite word vector or a composite vector;
a sequence determiner configured to calculate, in order from a first word of the entered text, a hidden vector for a t-th word by using a hidden vector calculated for a (t−1)-th word and a composite word vector or a composite vector output for the t-th word from the word coding unit; and
a sequence decoder configured to generate a prediction result of the output text from the hidden vector calculated for each word included in the entered text,
each word includes character N-grams, and the word coder includes:
a character N-gram vector acquirer configured to acquire, for each word included in the entered text, a vector representing each of the character N-grams included in the word;
a weight determiner configured to calculate a weight for a predetermined vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the word;
a character N-gram vector combiner configured to generate a composite vector obtained by combining the predetermined vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the word and the weight for the predetermined vector representing each of the character N-grams;
a word vector acquirer configured to acquire a vector representing the word; and
a word vector combiner configured to generate a composite word vector obtained by combining the vector representing the word and the composite vector generated by the character N-gram vector combiner.

### Claim 3 (depends on 2)

The analysis device according to claim 2, the device further comprising:
a parameter learner configured to learn a predetermined vector representing the character N-grams, a parameter used in calculation of the weight, a vector representing the word, and a parameter of each of the sequence coder and the sequence decoder on a basis of the prediction result of the output text by the analysis device for a learning text and a correct answer of the output text for the learning text.

### Claim 4

A word coding method in which a word coder converts an entered word into a composite vector by using a pre-learned neural network, the entered word including character N-grams, the method comprising:
converting at the word coder, wherein
acquiring, by a character N-gram vector acquirer, a predetermined vector representing each of the character N-grams included in a word;
determining, by a weight determiner, a weight for a vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the word; and
generating, by a character N-gram vector combiner a composite vector obtained by combining the predetermined vector representing each of the character N-grams on a basis of the predetermined vector representing each of the character N-grams included in the word and the weight for the predetermined vector representing each of the character N-grams.

### Claim 5

(canceled)

### Claim 6 (depends on 1)

The word coding device according to claim 1, the device further comprising:
a text receiver configured to receive a text, wherein the text includes the entered words.

### Claim 7 (depends on 1)

The word coding device according to claim 1, wherein the word coder includes a recurrent neural network.

### Claim 8 (depends on 1)

The word coding device according to claim 1, wherein the word coder uses a learnt recurrent neural network language model using the character N-gram vector.

### Claim 9 (depends on 1)

The word coding device according to claim 1, wherein the weight is associated with each dimension of the character N-gram vector.

### Claim 10 (depends on 1)

The word coding device according to claim 1, wherein the entered word represents a prefix of a word.

### Claim 11 (depends on 1)

The word coding device according to claim 1, wherein the entered word represents a suffix of a word.

### Claim 12 (depends on 2)

The analysis device according to claim 2, the device further comprising:
a text receiver configured to receive a text, wherein the text includes the entered words.

### Claim 13 (depends on 2)

The analysis device according to claim 2, wherein the word coder includes a recurrent neural network.

### Claim 14 (depends on 2)

The analysis device according to claim 2, wherein the word coder uses a learnt recurrent neural network language model using the character N-gram vector.

### Claim 15 (depends on 2)

The analysis device according to claim 2, wherein the entered word represents a prefix of a word.

### Claim 16 (depends on 2)

The analysis device according to claim 2, wherein the entered word represents a suffix of a word.

### Claim 17 (depends on 4)

The word coding method according to claim 4, the method further comprising:
receiving, by a text receiver, a text, wherein the text includes the entered words.

### Claim 18 (depends on 4)

The word coding method according to claim 4, wherein the word coder includes a recurrent neural network.

### Claim 19 (depends on 4)

The word coding method according to claim 4, wherein the word coder uses a learnt recurrent neural network language model using the character N-gram vector.

### Claim 20 (depends on 4)

The word coding method according to claim 4, wherein the entered word represents a prefix of a word.

### Claim 21 (depends on 4)

The word coding method according to claim 4, wherein the entered word represents a suffix of a word.
