# US 2014/0236577 A1 · Semantic Representations of Rare Words in a Neural Probabilistic Language Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0236577 A1
- Document ID: US20140236577A1
- Status: Published application
- Kind: A1
- Application number: 14/166,228
- Filed: 2014-01-28
- Published: 2014-08-21
- Earliest claimed: 2013-02-15
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Christopher Malon; Bing Bai
- Web page: https://patentgrep.com/patent/US20140236577A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/30: Semantic analysis
- G06N 3/02: Neural networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are disclosed for representing a word by extracting n-dimensions for the word from an original language model; if the word has been previously processed, use values previously chosen to define an (n+m) dimensional vector and otherwise randomly selecting m values to define the (n+m) dimensional vector; and applying the (n+m) dimensional vector to represent words that are not well-represented in the language model.

## Claims

### Claim 1

A method for representing a word, comprising:
extracting n-dimensions for the word from an original language model; and
if the word has been previously processed, use values previously chosen to define an (n+m) dimensional vector and otherwise randomly selecting m values to define the (n+m) dimensional vector.

### Claim 2 (depends on 1)

The method of claim 1, comprising applying the n-dimensional language vector for syntactic tagging tasks.

### Claim 3 (depends on 1)

The method of claim 1, comprising training outputs S(x,y)=(z0,z1) to minimize the cross-entropy function
h

(
(
z
0
,
z
1
)
,
j
)
=
-
log
(

z
j
e
z
0
+
e
z
1
)


for


j
=
0
,
1.
so that z0 and z1 estimate log likelihoods and a descendant relation is satisfied.

### Claim 4 (depends on 1)

The method of claim 1, comprising applying the (n+m) dimensional language vector to distinguish rare words.

### Claim 5 (depends on 1)

The method of claim 1, comprising answering free form questions using recursive neural network (RNN).

### Claim 6 (depends on 1)

The method of claim 1, comprising:
defining feature representations at every node of a parse trees of questions and supporting sentences, when applied recursively, starting with token vectors from a neural probabilistic language model; and
extracting answers to arbitrary natural language questions from supporting sentences.

### Claim 7 (depends on 1)

The method of claim 1, comprising training on a crowdsourced data set.

### Claim 8 (depends on 1)

The method of claim 1, comprising recursively classifying nodes of the parse tree of a supporting sentence.

### Claim 9 (depends on 1)

The method of claim 1, comprising using learned representations of words and syntax in a parse tree structure to answer free form questions about natural language text.

### Claim 10 (depends on 1)

The method of claim 1, comprising deciding to follow each parse tree node of a support sentence by classifying its RNN embedding together with those of siblings and a root node of the question, until reaching the tokens selected as the answer.

### Claim 11 (depends on 1)

The method of claim 1, comprising performing a co-training task for the RNN, on subtree recognition.

### Claim 12 (depends on 6)

The method of claim 6, wherein the co-training task for training the RNN that preserves deep structural information.

### Claim 13 (depends on 1)

The method of claim 1, comprising applying atop-down supervised method using continuous word features in parse trees to find an answer.

### Claim 14 (depends on 1)

The method of claim 1, wherein positively classified nodes are followed down the tree, and any positively classified terminal nodes become the tokens in the answer.

### Claim 15 (depends on 1)

The method of claim 1, wherein feature representations are dense vectors in a continuous feature space and for the terminal nodes, the dense vectors comprise word vectors in a neural probabilistic language model, and for interior nodes, the dense vectors are derived from children by recursive application of an autoencoder.

### Claim 16

A natural language system, comprising:
a processor to receive text and to represent a word;
computer code to extract n-dimensions for the word from an original language model; and
computer code to determine if the word has been previously processed, use values previously chosen to define an (n+m) dimensional vector and otherwise randomly selecting m values to define the (n+m) dimensional vector.

### Claim 17 (depends on 16)

The system of claim 16, comprising computer code for applying the n-dimensional language vector for syntactic tagging tasks.

### Claim 18 (depends on 16)

The system of claim 16, comprising computer code for comprising training outputs S(x,y)=(z0,z1) to minimize the cross-entropy function
h

(
(
z
0
,
z
1
)
,
j
)
=
-
log
(

z
j
e
z
0
+
e
z
1
)


for


j
=
0
,
1.
so that z0 and z1 estimate log likelihoods and a descendant relation is satisfied.

### Claim 19 (depends on 16)

The system of claim 16, comprising computer code for applying the (n+m) dimensional language vector to distinguish rare words.

### Claim 20 (depends on 16)

The system of claim 16, comprising computer code for answering free form questions using recursive neural network (RNN).
