# US 2024/0203003 A1 · Generation of Context-Aware Word Embedding Vectors for Given Semantic Properties of a Word Using Few Texts

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203003 A1
- Document ID: US20240203003A1
- Status: Published application
- Kind: A1
- Application number: 18/066,613
- Filed: 2022-12-15
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Accenture Global Solutions Limited
- Applicants: Accenture Global Solutions Limited
- Inventors: Anup Bera
- Web page: https://patentgrep.com/patent/US20240203003A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 11/60: Creating or editing images; Combining images with text
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This disclosure describes example implementations for generating context-dependent embedding vectors of words in the multi-dimensional embedding space based on a generic and non-domain-specific pretrained word embeddings. Such an implementation requires no domain specific training corpus and is capable of generating context-dependent embedding vectors of multi-semantic words using a few contextual texts. Such an implementation thus provides an efficient way to generate a library of multiple domain specific embedding vectors for multi-semantic words without any domain-specific training process. Other example embodiments further apply the principles of the context-dependent word embedding generation to a text-to-image application.

## Claims

### Claim 1

An automatic context-aware embedding generation system, comprising:
a pretrained context-aware natural language processing (NLP) circuitry for generating a sentence or higher-level embedding vector of a textual input expression in a multi-dimensional embedding space, the pretrained context-aware NLP circuitry comprising a neural network having at least one hidden layer;
a context-aware embedding generator circuitry for generating embedding vectors at a word-level; and
a datastore for storing context-aware word-level embedding vectors,
wherein:
the context-aware embedding generator circuitry is configured to receive the textual input expression and request the pretrained context-aware NLP circuitry to process the textual input expression;
the pretrained context-aware NLP circuitry is configured to automatically parse the textual input expression into a plurality of input tokens, automatically generate context-aware information items in the at least one hidden layer corresponding to the plurality of input tokens, and automatically generate the sentence or higher-level embedding vector of the input textual expression in the multi-dimensional embedding space;
the context-aware embedding generator circuitry is further configured to automatically aggregate the context-aware information items from the at least one hidden layer to derive a set of word-level context-aware embedding vectors in the multi-dimensional embedding space for the plurality of input tokens; and
the context-aware embedding generator circuitry is further configured to store the set of word-level context-aware embedding vectors in the multi-dimensional embedding space for the plurality of input tokens in the datastore, wherein at least one word in the datastore corresponds to two or more word-level context-aware embedding vectors.

### Claim 2 (depends on 1)

The automatic context-aware embedding generation system of claim 1, wherein the context-aware embedding generator circuitry is configured to:
obtain a plurality of reference embedding vectors of the plurality of input tokens in the multi-dimensional embedding space from a pretrained word-level reference embedding library; and
mix the plurality of reference embedding vectors to generate the context-aware information items for the plurality of input tokens in the at least one hidden layer.

### Claim 3 (depends on 2)

The automatic context-aware embedding generation system of claim 2, wherein the neural network of the context-aware embedding generator circuitry comprises at least a natural language processing model based on bidirectional encoder representations from Transformers (BERT).

### Claim 4 (depends on 2)

The automatic context-aware embedding generation system of claim 2, wherein the neural network of the context-aware embedding generator circuitry comprises a combination of at least two separate NLP models.

### Claim 5 (depends on 1)

The automatic context-aware embedding generation system of claim 1, further comprising a text-to-image generator circuitry and a display screen, wherein the text-to-image generator circuitry is configured to:
automatically extract a subject token and a property token from the plurality of input tokens,
automatically derive from the property token a property embedding vector in the multi-dimensional embedding space;
automatically generate a textual paragraph comprising a plurality of sentences based on the subject token;
automatically obtain, from the plurality of sentences using the context-aware embedding generator circuitry, a plurality of embedding-vector-property-token pairs, each of the plurality of embedding-vector-property-token pairs comprising:
a context-aware embedding vector of the subject token in the multi-dimensional embedding space representing a context-aware semantics of the subject token within one of the plurality of sentences; and
a property word associated with the context-aware embedding vector of the subject token in a context of a corresponding sentence of the plurality of sentences;
automatically generate a realism measure of the textual input expression based on the plurality of embedding-vector-property-token pairs and the property embedding vector of the property token;
when the realism measure is above a predefined threshold level, automatically generate a visual image from the textual input expression; and
display the visual image on the display screen.

### Claim 6 (depends on 5)

The automatic context-aware embedding generation system of claim 5, wherein the property embedding vector is derived from a pretrained word-level reference embedding library associated with the multi-dimensional embedding space or derived from the textual input expression using the pretrained context-aware embedding generator circuitry.

### Claim 7 (depends on 6)

The automatic context-aware embedding generation system of claim 6, wherein the plurality of sentences of the textual paragraph each comprises the subject token and a characteristic property associated with the subject token.

### Claim 8 (depends on 7)

The automatic context-aware embedding generation system of claim 7, wherein the context-aware embedding generator circuitry is configured to:
obtain a plurality of reference embedding vectors of the plurality of input tokens in the multi-dimensional embedding space from the pretrained word-level reference embedding library; and
mix the plurality of reference embedding vectors to generate the context-aware information items for the plurality of input tokens in the at least one hidden layer.

### Claim 9 (depends on 8)

The automatic context-aware embedding generation system of claim 8, wherein the neural network of the context-aware embedding generator circuitry comprises at least a natural language processing model based on BERT.

### Claim 10 (depends on 8)

The automatic context-aware embedding generation system of claim 8, wherein the neural network of the context-aware embedding generator circuitry comprises a combination of at least two separate NLP models.

### Claim 11 (depends on 5)

The automatic context-aware embedding generation system of claim 5, wherein to automatically generate the realism measure of the textual input expression comprises to:
determine whether the property token matches any of the property words in the plurality of embedding-vector-property-token pairs; and
determine that the realism measure is above the predefined threshold level when the property token matches any of the property words in the plurality of embedding-vector-property-token pairs.

### Claim 12 (depends on 11)

The automatic context-aware embedding generation system of claim 11, wherein to automatically generate the realism measure of the textual input expression further comprises to:
compare the property embedding vector to the context-aware embedding vectors of the subject token in the plurality of embedding-vector-property-token pairs when it is determined that the property token does not match any of the property words in the plurality of embedding-vector-property-token pairs; and
determine whether the realism measure is above the predefined threshold level by evaluating a similarity of the property embedding vector and any of the context-aware embedding vectors of the subject token in the plurality of embedding-vector-property-token pairs.

### Claim 13 (depends on 12)

The automatic context-aware embedding generation system of claim 12, wherein the similarity is evaluated according to a distance between the property embedding vector and any of the context-aware embedding vectors of the subject token in the plurality of embedding-vector-property-token pairs in the multi-dimensional embedding space.

### Claim 14

An automatic context-aware embedding generation method, comprising:
receiving a textual input expression;
automatically parsing the textual input expression into a plurality of input tokens;
automatically generating context-aware information items corresponding to the plurality of input tokens in at least one hidden layer of a pretrained context-aware natural language processing (NLP) model configured to generate sentence or higher-level embedding vectors in a multi-dimensional embedding space;
automatically generating a sentence-level embedding vector of the textual input expression in the multi-dimensional embedding space;
automatically aggregating the context-aware information items from the at least one hidden layer to derive a set of word-level context-aware embedding vectors in the multi-dimensional embedding space for the plurality of input tokens; and
storing the set of word-level context-aware embedding vectors in the multi-dimensional embedding space for the plurality of input tokens in an embedding datastore, wherein at least one word in the embedding datastore corresponds to two or more word-level context-aware embedding vectors.

### Claim 15 (depends on 14)

The automatic context-aware embedding generation method of claim 14, further comprising:
obtaining a plurality of reference embedding vectors of the plurality of input tokens in the multi-dimensional embedding space from a pretrained word-level reference embedding library; and
mixing the plurality of reference embedding vectors to generate the context-aware information items for the plurality of input tokens in the at least one hidden layer.

### Claim 16 (depends on 14)

The automatic context-aware embedding generation method of claim 14, further comprising:
automatically extracting a subject token and a property token from the plurality of input tokens,
automatically deriving from the property token a property embedding vector in the multi-dimensional embedding space;
automatically generating a textual paragraph comprising a plurality of sentences based on the subject token;
automatically obtaining, from the plurality of sentences using the pretrained context-aware NLP model, a plurality of embedding-vector-property-token pairs, each of the plurality of embedding-vector-property-token pairs comprising:
a context-aware embedding vector of the subject token in the multi-dimensional embedding space representing a context-aware semantics of the subject token within one of the plurality of sentences; and
a property word associated with the context-aware embedding vector of the subject token in a context of a corresponding sentence of the plurality of sentences;
automatically generating a realism measure of the textual input expression based on the plurality of embedding-vector-property-token pairs and the property embedding vector of the property token;
when the realism measure is above a predefined threshold level, automatically generating a visual image from the textual input expression; and
displaying the visual image on a display screen.

### Claim 17 (depends on 16)

The automatic context-aware embedding generation method of claim 16, wherein the property embedding vector is derived from a pretrained word-level reference embedding library associated with the multi-dimensional embedding space or derived from the textual input expression using the pretrained context-aware NLP model.

### Claim 18 (depends on 16)

The automatic context-aware embedding generation method of claim 16, further comprising:
obtaining a plurality of reference embedding vectors of the plurality of input tokens in the multi-dimensional embedding space from a pretrained word-level reference embedding library; and
mixing the plurality of reference embedding vectors to generate the context-aware information items for the plurality of input tokens in the at least one hidden layer.

### Claim 19 (depends on 16)

The automatic context-aware embedding generation method of claim 16, wherein automatically generating the realism measure of the textual input expression comprises:
determining whether the property token matches any of the property words in the plurality of embedding-vector-property-token pairs; and
determining that the realism measure is above the predefined threshold level when the property token matches any of the property words in the plurality of embedding-vector-property-token pairs.

### Claim 20 (depends on 19)

The automatic context-aware embedding generation method of claim 19, wherein automatically generating the realism measure of the textual input expression further comprises:
comparing the property embedding vector to the context-aware embedding vectors of the subject token in the plurality of embedding-vector-property-token pairs when it is determined that the property token does not match any of the property words in the plurality of embedding-vector-property-token pairs; and
determining whether the realism measure is above the predefined threshold level by evaluating a similarity of the property embedding vector and any of the context-aware embedding vectors of the subject token in the plurality of embedding-vector-property-token pairs.