# US 2023/0274098 A1 · Meaning and Sense Preserving Textual Encoding and Embedding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274098 A1
- Document ID: US20230274098A1
- Status: Published application
- Kind: A1
- Application number: 17/682,028
- Filed: 2022-02-28
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Tanveer Syeda-Mahmood
- Web page: https://patentgrep.com/patent/US20230274098A1

## CPC

- G06F 40/56: Natural language generation
- G06F 40/247: Thesauruses; Synonyms
- G06F 40/30: Semantic analysis
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Mechanisms for implementing a text encoder and text encoder operations are provided. A contrastive machine learning training operation trains an encoder of a machine learning computer model, to learn a sense and similarity preserving embedding that operates to encode input natural language text data to generate encoded natural language text data based on a sense attribute of one or more terms in the input natural language text data. The contrastive machine learning training operation learns to separate positive samples in training data from negative samples in the training data. The trained encoder processes a term specified in an input natural language text to generate an encoded natural language text based on the learned embedding and inputs, to a downstream computing system, the encoded natural language text to cause the downstream computing system to perform a computer natural language processing operation based on the embedding.

## Claims

### Claim 1

A method, in a data processing system comprising at least one processor and at least one memory, the at least one memory comprising instructions that are executed by the at least on processor to cause the at least one processor to be specifically configured to implement a text encoder that encodes natural language text data input at least by:
training, by a contrastive machine learning training operation, an encoder of a machine learning computer model, to learn a sense and similarity preserving embedding, wherein the sense and similarity preserving embedding operates to encode input natural language text data to generate encoded natural language text data based on a sense attribute of one or more terms in the input natural language text data, and wherein the contrastive machine learning training operation operates to learn to separate positive samples in training data from negative samples in the training data;
processing, by the trained encoder computer model, a first term specified in an input natural language text to generate an encoded natural language text based on the learned sense and similarity preserving embedding; and
inputting, to a downstream computing system, the encoded natural language text, to cause the downstream computing system to perform a computer natural language processing operation on the encoded natural language text data based on the sense and similarity preserving embedding.

### Claim 2 (depends on 1)

The method of claim 1, wherein training the encoder by the contrastive machine learning training operation comprises, for each term in a vocabulary ontology data structure:
processing the vocabulary ontology data structure to represent the term as a first tuple data structure in which the first tuple data structure specifies a sense attribute of the corresponding term; and
generating, for the term, at least one similarity list based on at least the sense attribute specified in the first tuple and a sense attribute specified in second tuples of other terms in the vocabulary ontology data structure; and wherein training the encoder further comprises:
inputting training data into the encoder, wherein the training data comprises, for each specified term in the vocabulary ontology data structure, first samples corresponding to generated similarity lists for the specified term, and second samples comprising negative samples specifying other terms in the vocabulary ontology data structure that are not similar to the specified term; and
performing, by a decoder of the machine learning computer model, the contrastive learning operation based on the first samples and second samples and a contrastive loss function.

### Claim 3 (depends on 2)

The method of claim 2, wherein one or more terms in the vocabulary ontology data structure have a plurality of tuples, each tuple corresponding to a different sense attribute of the same term.

### Claim 4 (depends on 3)

The method of claim 3, wherein different terms in the vocabulary ontology data structure have a different number of sense attributes and a different number of tuples.

### Claim 5 (depends on 2)

The method of claim 2, wherein the first tuple data structure further comprises a first data value specifying the term, a second data value specifying a part of speech attribute, a third data value specifying the sense attribute, and a fourth data value that specifies a lemma attribute.

### Claim 6 (depends on 2)

The method of claim 2, wherein inputting the training data into the encoder the training comprises an adaptive batching operation that divides the training data into batches comprising pairings of vector representations of the specified terms, in the vocabulary ontology data structure, with corresponding similarity lists of the specified terms, wherein each batch has a maximum size and comprises pairings whose members are disjointed and a number of pairings whose total size is equal to or less than the maximum size.

### Claim 7 (depends on 2)

The method of claim 2, wherein generating the similarity lists comprises traversing the ontology data structure and, for each specified term in the ontology data structure, comprises generating, for each other term in the ontology data structure, a similarity metric value based on a depth of the other term in the ontology data structure and a least common ancestor of the other term and the specified term.

### Claim 8 (depends on 2)

The method of claim 2, wherein the at least one similarity list comprises other terms in the vocabulary ontology data structure whose similarity metric is equal to or above a specified threshold and does not include other terms in the vocabulary ontology data structure whose similarity metric is below the specified threshold.

### Claim 9 (depends on 1)

The method of claim 1, wherein the sense and similarity preserving embedding represents the sense attribute as a numeric formulation in a distance space.

### Claim 10 (depends on 1)

The method of claim 1, wherein the input natural language text data comprises a query term, and wherein the computer natural language processing operation is an operation to identify similar terms present in a vocabulary ontology data structure that are similar to the query term.

### Claim 11

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on data processing system, causes the data processing system to:
train, by a contrastive machine learning training operation, an encoder of a machine learning computer model, to learn a sense and similarity preserving embedding, wherein the sense and similarity preserving embedding operates to encode input natural language text data to generate encoded natural language text data based on a sense attribute of one or more terms in the input natural language text data, and wherein the contrastive machine learning training operation operates to learn to separate positive samples in training data from negative samples in the training data;
process, by the trained encoder computer model, a first term specified in an input natural language text to generate an encoded natural language text based on the learned sense and similarity preserving embedding; and
input, to a downstream computing system, the encoded natural language text, to cause the downstream computing system to perform a computer natural language processing operation on the encoded natural language text data based on the sense and similarity preserving embedding.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein training the encoder by the contrastive machine learning training operation comprises, for each term in a vocabulary ontology data structure:
processing the vocabulary ontology data structure to represent the term as a first tuple data structure in which the first tuple data structure specifies a sense attribute of the corresponding term; and
generating, for the term, at least one similarity list based on at least the sense attribute specified in the first tuple and a sense attribute specified in second tuples of other terms in the vocabulary ontology data structure; and wherein training the encoder further comprises:
inputting training data into the encoder, wherein the training data comprises, for each specified term in the vocabulary ontology data structure, first samples corresponding to generated similarity lists for the specified term, and second samples comprising negative samples specifying other terms in the vocabulary ontology data structure that are not similar to the specified term; and
performing, by a decoder of the machine learning computer model, the contrastive learning operation based on the first samples and second samples and a contrastive loss function.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein one or more terms in the vocabulary ontology data structure have a plurality of tuples, each tuple corresponding to a different sense attribute of the same term.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein different terms in the vocabulary ontology data structure have a different number of sense attributes and a different number of tuples.

### Claim 15 (depends on 12)

The computer program product of claim 12, wherein the first tuple data structure further comprises a first data value specifying the term, a second data value specifying a part of speech attribute, a third data value specifying the sense attribute, and a fourth data value that specifies a lemma attribute.

### Claim 16 (depends on 12)

The computer program product of claim 12, wherein inputting the training data into the encoder the training comprises an adaptive batching operation that divides the training data into batches comprising pairings of vector representations of the specified terms, in the vocabulary ontology data structure, with corresponding similarity lists of the specified terms, wherein each batch has a maximum size and comprises pairings whose members are disjointed and a number of pairings whose total size is equal to or less than the maximum size.

### Claim 17 (depends on 12)

The computer program product of claim 12, wherein generating the similarity lists comprises traversing the ontology data structure and, for each specified term in the ontology data structure, comprises generating, for each other term in the ontology data structure, a similarity metric value based on a depth of the other term in the ontology data structure and a least common ancestor of the other term and the specified term.

### Claim 18 (depends on 12)

The computer program product of claim 12, wherein the at least one similarity list comprises other terms in the vocabulary ontology data structure whose similarity metric is equal to or above a specified threshold and does not include other terms in the vocabulary ontology data structure whose similarity metric is below the specified threshold.

### Claim 19 (depends on 11)

The computer program product of claim 11, wherein the sense and similarity preserving embedding represents the sense attribute as a numeric formulation in a distance space.

### Claim 20

An apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to:
train, by a contrastive machine learning training operation, an encoder of a machine learning computer model, to learn a sense and similarity preserving embedding, wherein the sense and similarity preserving embedding operates to encode input natural language text data to generate encoded natural language text data based on a sense attribute of one or more terms in the input natural language text data, and wherein the contrastive machine learning training operation operates to learn to separate positive samples in training data from negative samples in the training data;
process, by the trained encoder computer model, a first term specified in an input natural language text to generate an encoded natural language text based on the learned sense and similarity preserving embedding; and
input, to a downstream computing system, the encoded natural language text, to cause the downstream computing system to perform a computer natural language processing operation on the encoded natural language text data based on the sense and similarity preserving embedding.
