# US 2017/0286397 A1 · Predictive Embeddings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286397 A1
- Document ID: US20170286397A1
- Status: Published application
- Kind: A1
- Application number: 15/085,973
- Filed: 2016-03-30
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Daniel P. Gonzalez
- Web page: https://patentgrep.com/patent/US20170286397A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/295: Named entity recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An approach is provided in which an information handling system detects an unknown word in a sentence and generates a context embedding using known words in proximity to the unknown word in the sentence. Next, the information handling system creates a predictive embedding of to the unknown word based upon the context embedding. The predictive embedding corresponds to an embedding area of the unknown word without specifying the unknown word. In turn, the information handling system utilizes the predictive embedding to generate natural language processing results corresponding to the sentence.

## Claims

### Claim 1

A method implemented by an information handling system that includes a memory and a processor, the method comprising:
generating a context embedding corresponding to a plurality of known words in a sentence that are in proximity to an unknown word in the sentence;
creating a predictive embedding corresponding to the unknown word based on the context embedding, wherein the predictive embedding corresponds to an embedding area of the unknown word without specifying the unknown word; and
utilizing the predictive embedding to generate one or more natural language processing results corresponding to the sentence.

### Claim 2 (depends on 1)

The method of claim 1 wherein the generation of the context embedding further comprises:
detecting the unknown word in the sentence;
creating a context from the plurality of known words;
concatenating a plurality of word embeddings corresponding to the plurality of known words, the concatenating resulting in the context embedding.

### Claim 3 (depends on 1)

The method of claim 1 wherein the creating of the predictive embedding is performed by a predictive embedding model and, prior to the creating of the predictive embedding, the method further comprises:
training the predictive embedding model using a training sentence that includes a plurality of training words, wherein the training further comprises:
randomly selecting one of the plurality of training words;
generating a training context using a set of the plurality of training words in proximity to the randomly selected training word;
generating a training context embedding corresponding to the training context using a set of training word embeddings corresponding to the set training words included in the training context; and
training the predictive embedding model using the training context embedding.

### Claim 4 (depends on 1)

The method of claim 1 wherein the predictive embedding is a predictive embedding vector comprising a plurality of numeric coordinates, and wherein each of the plurality of numeric coordinates corresponds to at least one of a plurality of features of the unknown word.

### Claim 5 (depends on 4)

The method of claim 4 wherein the predictive embedding vector points to the embedding area that comprises a plurality of similar words that are similar in meaning to the unknown word, and wherein the embedding area fails to include the unknown word.

### Claim 6 (depends on 1)

The method of claim 1 further comprising:
concatenating the predictive embedding with a plurality of word embeddings corresponding to the plurality of known words in the sentence, resulting in a concatenated predictive embedding; and
utilizing the concatenated predictive embedding in the generation of the one or more natural language processing results.

### Claim 7 (depends on 1)

The method of claim 1 wherein at least one of the one or more natural language processing results is based upon a post-processing task selected from the group consisting of a sentiment analysis task, a syntactic parsing task, and a named entity recognition task.

### Claim 8

An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors; and
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
generating a context embedding corresponding to a plurality of known words in a sentence that are in proximity to an unknown word in the sentence;
creating a predictive embedding corresponding to the unknown word based on the context embedding, wherein the predictive embedding corresponds to an embedding area of the unknown word without specifying the unknown word; and
utilizing the predictive embedding to generate one or more natural language processing results corresponding to the sentence.

### Claim 9 (depends on 8)

The information handling system of claim 8 wherein at least one of the one or more processors perform additional actions comprising:
detecting the unknown word in the sentence;
creating a context from the plurality of known words;
concatenating a plurality of word embeddings corresponding to the plurality of known words, the concatenating resulting in the context embedding.

### Claim 10 (depends on 8)

The information handling system of claim 8 wherein the creating of the predictive embedding is performed by a predictive embedding model and, prior to the creating of the predictive embedding, and wherein at least one of the one or more processors perform additional actions comprising:
training the predictive embedding model using a training sentence that includes a plurality of training words, wherein the training further comprises:
randomly selecting one of the plurality of training words;
generating a training context using a set of the plurality of training words in proximity to the randomly selected training word;
generating a training context embedding corresponding to the training context using a set of training word embeddings corresponding to the set training words included in the training context; and
training the predictive embedding model using the training context embedding.

### Claim 11 (depends on 8)

The information handling system of claim 8 wherein the predictive embedding is a predictive embedding vector comprising a plurality of numeric coordinates, and wherein each of the plurality of numeric coordinates corresponds to at least one of a plurality of features of the unknown word.

### Claim 12 (depends on 11)

The information handling system of claim 11 wherein the predictive embedding vector points to the embedding area that comprises a plurality of similar words that are similar in meaning to the unknown word, and wherein the embedding area fails to include the unknown word.

### Claim 13 (depends on 8)

The information handling system of claim 8 wherein at least one of the one or more processors perform additional actions comprising:
concatenating the predictive embedding with a plurality of word embeddings corresponding to the plurality of known words in the sentence, resulting in a concatenated predictive embedding; and
utilizing the concatenated predictive embedding in the generation of the one or more natural language processing results.

### Claim 14 (depends on 8)

The information handling system of claim 8 wherein at least one of the one or more natural language processing results is based upon a post-processing task selected from the group consisting of a sentiment analysis task, a syntactic parsing task, and a named entity recognition task.

### Claim 15

A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:
generating a context embedding corresponding to a plurality of known words in a sentence that are in proximity to an unknown word in the sentence;
creating a predictive embedding corresponding to the unknown word based on the context embedding, wherein the predictive embedding corresponds to an embedding area of the unknown word without specifying the unknown word; and
utilizing the predictive embedding to generate one or more natural language processing results corresponding to the sentence.

### Claim 16 (depends on 15)

The computer program product of claim 15 wherein the information handling system performs additional actions comprising:
detecting the unknown word in the sentence;
creating a context from the plurality of known words;
concatenating a plurality of word embeddings corresponding to the plurality of known words, the concatenating resulting in the context embedding.

### Claim 17 (depends on 15)

The computer program product of claim 15 wherein the creating of the predictive embedding is performed by a predictive embedding model and, prior to the creating of the predictive embedding, and wherein the information handling system performs additional actions comprising:
training the predictive embedding model using a training sentence that includes a plurality of training words, wherein the training further comprises:
randomly selecting one of the plurality of training words;
generating a training context using a set of the plurality of training words in proximity to the randomly selected training word;
generating a training context embedding corresponding to the training context using a set of training word embeddings corresponding to the set training words included in the training context; and
training the predictive embedding model using the training context embedding.

### Claim 18 (depends on 15)

The computer program product of claim 15 wherein the predictive embedding is a predictive embedding vector comprising a plurality of numeric coordinates, and wherein each of the plurality of numeric coordinates corresponds to at least one of a plurality of features of the unknown word.

### Claim 19 (depends on 18)

The computer program product of claim 18 wherein the predictive embedding vector points to the embedding area that comprises a plurality of similar words that are similar in meaning to the unknown word, and wherein the embedding area fails to include the unknown word.

### Claim 20 (depends on 15)

The computer program product of claim 15 wherein the information handling system performs additional actions comprising:
concatenating the predictive embedding with a plurality of word embeddings corresponding to the plurality of known words in the sentence, resulting in a concatenated predictive embedding; and
utilizing the concatenated predictive embedding in the generation of the one or more natural language processing results.