# US 2024/0202176 A1 · Self contrastive decorrelation based training of machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202176 A1
- Document ID: US20240202176A1
- Status: Published application
- Kind: A1
- Application number: 18/081,920
- Filed: 2022-12-15
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Sap SE
- Applicants: Sap SE
- Inventors: Tassilo Klein; Moin Nabi
- Web page: https://patentgrep.com/patent/US20240202176A1

## CPC

- G06F 16/2237: {Vectors, bitmaps or matrices}
- G06F 16/243: {Natural language query formulation}
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for training a machine learning model using self-contrastive decorrelation is provided. The method comprises training a machine learning model by receiving a sentence including text, performing a first encoding operation on the sentence, performing a second encoding operation on the sentence, mapping the first vector representation on which a first augmentation operation is performed to a first high dimensional vector representation and the second vector representation on which a first augmentation operation is performed to a second high dimensional vector representation, generating a correlation matrix using the first high dimensional vector representation and the second high dimensional vector representation, and performing a decorrelation operation on the correlation matrix. The method includes receiving, by the trained machine learning model, an query that includes a target sentence, and outputting, using the trained machine learning model, a result sentence that satisfies a similarity metric relative to the target sentence.

## Claims

### Claim 1

A computer-implemented method comprising:
training a machine learning model by:
receiving a sentence including a plurality of text;
performing a first encoding operation on the sentence, the first encoding operation comprising:
generating a first vector representation of the sentence, the first vector representation including numerical elements representing the plurality of text, and
performing a first augmentation operation on the first vector representation, the first augmentation operation including masking one or more of the numerical elements of the first vector representation in accordance with a first dropout ratio;
performing a second encoding operation on the sentence, the second encoding operation comprising:
generating a second vector representation of the sentence, the second vector representation including additional numerical elements representing the plurality of text, and
performing a second augmentation operation on the second vector representation, the second augmentation operation including masking one or more of the numerical elements of the second vector representation in accordance with a second dropout ratio, wherein the second dropout ratio is larger than the first dropout ratio;
mapping the first vector representation on which the first augmentation operation is performed to a first high dimensional vector representation and the second vector representation on which the first augmentation operation is performed to a second high dimensional vector representation;
generating a correlation matrix using the first high dimensional vector representation and the second high dimensional vector representation;
performing a decorrelation operation on the correlation matrix;
receiving, by the machine learning model that is trained, a query that includes a target sentence; and
outputting, using the machine learning model that is trained, a result sentence that satisfies a similarity metric relative to the target sentence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the first vector representation of the sentence on which the first augmentation operation is performed satisfies a first similarity threshold relative to a sentiment of the sentence including the plurality of text.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the second vector representation of the sentence on which the second augmentation operation is performed satisfies a second similarity threshold relative to the sentiment of the sentence including the plurality of text.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the second threshold similarity is less than the first similarity threshold.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the performing of the decorrelation operation on the correlation matrix is based on implementation of a loss objective decorrelation function.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the loss objective decorrelation function comprises an augmentation invariance term, feature redundancy management term, and a hyper-parameter.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the performing of the decorrelation operation based on the implementation of the loss objective decorrelation function comprises:
multiplying the hyper-parameter with the feature redundancy management term; and
summing the augmentation invariance term with the multiplying of the hyper-parameter with the feature redundancy management term.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving an image;
performing a third encoding operation on the image, the first encoding operation comprising:
generating a third vector representation of the image, the third vector representation including numerical elements representing the image, and
performing a third augmentation operation on the third vector representation, the third augmentation operation including masking one or more of the numerical elements of the third vector representation in accordance with a third dropout ratio;
performing a fourth encoding operation on the image, the fourth encoding operation comprising:
generating a fourth vector representation of the image, the fourth vector representation including additional numerical elements representing the image, and
performing a fourth augmentation operation on the fourth vector representation, the fourth augmentation operation including masking one or more of the numerical elements of the fourth vector representation in accordance with a fourth dropout ratio, wherein the fourth dropout ratio is larger than the third dropout ratio;
mapping the third vector representation on which the third augmentation operation is performed to a third high dimensional vector representation and the fourth vector representation on which the fourth augmentation operation is performed to a fourth high dimensional vector representation;
generating an additional correlation matrix using the third high dimensional vector representation and the fourth high dimensional vector representation; and
performing an additional decorrelation operation on the additional correlation matrix.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the performing of the additional decorrelation operation on the additional correlation matrix is based on implementation of a loss-objective decorrelation function.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the loss objective decorrelation function comprises an augmentation invariance term, feature redundancy management term, and a hyper-parameter.

### Claim 11

A system comprising:
at least one data processor; and
at least one memory storing instructions, which when executed by the at least one data processor, cause operations comprising:
training a machine learning model by:
receiving a sentence including a plurality of text;
performing a first encoding operation on the sentence, the first encoding operation comprising:
generating a first vector representation of the sentence, the first vector representation including numerical elements representing the plurality of text, and
performing a first augmentation operation on the first vector representation, the first augmentation operation including masking one or more of the numerical elements of the first vector representation in accordance with a first dropout ratio;
performing a second encoding operation on the sentence, the second encoding operation comprising:
generating a second vector representation of the sentence, the second vector representation including additional numerical elements representing the plurality of text, and
performing a second augmentation operation on the second vector representation, the second augmentation operation including masking one or more of the numerical elements of the second vector representation in accordance with a second dropout ratio, wherein the second dropout ratio is larger than the first dropout ratio;
mapping the first vector representation on which the first augmentation operation is performed to a first high dimensional vector representation and the second vector representation on which the first augmentation operation is performed to a second high dimensional vector representation;
generating a correlation matrix using the first high dimensional vector representation and the second high dimensional vector representation;
performing a decorrelation operation on the correlation matrix;
receiving, by the machine learning model that is trained, a query that includes a target sentence; and
outputting, using the machine learning model that is trained, a result sentence that satisfies a similarity metric relative to the target sentence.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first vector representation of the sentence on which the first augmentation operation is performed satisfies a first similarity threshold relative to a sentiment of the sentence including the plurality of text.

### Claim 13 (depends on 12)

The system of claim 12, wherein the second vector representation of the sentence on which the second augmentation operation is performed satisfies a second similarity threshold relative to the sentiment of the sentence including the plurality of text.

### Claim 14 (depends on 13)

The system of claim 13, wherein the second similarity threshold is less than the first similarity threshold.

### Claim 15 (depends on 11)

The system of claim 11, wherein the performing of the decorrelation operation on the correlation matrix is based on implementation of a loss objective decorrelation function.

### Claim 16 (depends on 15)

The system of claim 15, wherein the loss objective decorrelation function comprises an augmentation invariance term, feature redundancy management term, and a hyper-parameter.

### Claim 17 (depends on 16)

The system of claim 16, wherein one of the operations of the performing of the decorrelation operation based on the implementation of the loss objective decorrelation function comprises:
multiplying the hyper-parameter with the feature redundancy management term; and
summing the augmentation invariance term with the multiplying of the hyper-parameter with the feature redundancy management term.

### Claim 18 (depends on 11)

The system of claim 11, wherein the operations further comprising:
receiving an image;
performing a third encoding operation on the image, the first encoding operation comprising:
generating a third vector representation of the image, the third vector representation including numerical elements representing the image, and
performing a third augmentation operation on the third vector representation, the third augmentation operation including masking one or more of the numerical elements of the third vector representation in accordance with a third dropout ratio;
performing a fourth encoding operation on the image, the fourth encoding operation comprising:
generating a fourth vector representation of the image, the fourth vector representation including additional numerical elements representing the image, and performing a fourth augmentation operation on the fourth vector representation, the fourth augmentation operation including masking one or more of the numerical elements of the fourth vector representation in accordance with a fourth dropout ratio, wherein the fourth dropout ratio is larger than the third dropout ratio;
mapping the third vector representation on which the third augmentation operation is performed to a third high dimensional vector representation and the fourth vector representation on which the fourth augmentation operation is performed to a fourth high dimensional vector representation;
generating an additional correlation matrix using the third high dimensional vector representation and the fourth high dimensional vector representation; and
performing an additional decorrelation operation on the additional correlation matrix.

### Claim 19 (depends on 18)

The system of claim 18, wherein the performing of the additional decorrelation operation on the additional correlation matrix is based on implementation of a loss-objective decorrelation function.

### Claim 20

A non-transitory computer-readable storage medium comprising programming code, which when executed by at least one data processor, causes operations comprising:
training a machine learning model by:
receiving a sentence including a plurality of text;
performing a first encoding operation on the sentence, the first encoding operation comprising:
generating a first vector representation of the sentence, the first vector representation including numerical elements representing the plurality of text, and
performing a first augmentation operation on the first vector representation, the first augmentation operation including masking one or more of the numerical elements of the first vector representation in accordance with a first dropout ratio;
performing a second encoding operation on the sentence, the second encoding operation comprising:
generating a second vector representation of the sentence, the second vector representation including additional numerical elements representing the plurality of text, and
performing a second augmentation operation on the second vector representation, the second augmentation operation including masking one or more of the numerical elements of the second vector representation in accordance with a second dropout ratio, wherein the second dropout ratio is larger than the first dropout ratio;
mapping the first vector representation on which the first augmentation operation is performed to a first high dimensional vector representation and the second vector representation on which the first augmentation operation is performed to a second high dimensional vector representation;
generating a correlation matrix using the first high dimensional vector representation and the second high dimensional vector representation;
performing a decorrelation operation on the correlation matrix;
receiving, by the machine learning model that is trained, a query that includes a target sentence; and
outputting, using the machine learning model that is trained, a result sentence that satisfies a similarity metric relative to the target sentence.
