# US 12,603,146 B2 · Multimodal domain embeddings via contrastive learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,603,146 B2
- Document ID: US12603146B2
- Status: Granted
- Kind: B2
- Application number: 17/410,701
- Filed: 2021-08-24
- Published: 2023-03-02
- Granted: 2026-04-14
- Earliest claimed: 2021-08-24
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Zhihui Guo; Pramod Kumar Sharma; Liang Du; Robin Abraham
- Web page: https://patentgrep.com/patent/US12603146B2

## CPC

- G16B 15/00: ICT specially adapted for analysing two-dimensional [2D] or three-dimensional [3D] molecular structures, e.g. structural or functional relations or structure alignment
- G16C 20/70: Machine learning, data mining or chemometrics
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G16H 70/40: relating to drugs, e.g. their side effects or intended usage
- G06N 3/048: Activation functions
- G16C 20/30: Prediction of properties of chemical compounds, compositions or mixtures
- G16C 20/40: Searching chemical structures or physicochemical data
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16C: COMPUTATIONAL CHEMISTRY; CHEMOINFORMATICS; COMPUTATIONAL MATERIALS SCIENCE
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Systems and methods are provided for building and training machine learning models configured to generate in-domain embeddings and perform multimodal analysis inside the same domain. The models include a first encoder trained to receive input from one or more entities represented in a first modality and to encode the one or more entities in the first modality, such that the first encoder is configured to output a first set of embeddings. The models also include a second encoder trained to receive input from one or more entities represented in the second modality and to encode the one or more entities in the second modality, such that the second encoder is configured to output a second set of embeddings. The models also include a projection layer configured to project the first set of embeddings and the second set of embeddings to a shared contrastive space.

## Claims

### Claim 1

A computing system configured to build a machine learning model that is trainable to perform in-domain embedding analysis, the computing system comprising:
one or more processors; and
one or more hardware storage devices storing one or more computer-readable instructions that are executable by the one or more processors to cause the computing system to:
access a first encoder of the machine learning model;
tokenize a first representation of an entity, resulting in generation of a first set of tokens for the entity, wherein the entity is represented in a first modality by the first representation;
embed the first set of tokens into a first numeric matrix representation;
generate a first positional encoding for the first set of tokens, wherein the first positional encoding preserves a first order of the tokens included in the first set of tokens;
provide, to the first encoder, the first numeric matrix representation and the first positional encoding;
cause the first encoder to encode the first numeric matrix representation and the first positional encoding, such that the first encoder is configured to output a first set of embeddings;
access a second encoder of the machine learning model; trained to receive as input one or more entities represented in a second modality and
tokenize a second representation of the entity, resulting in generation of a second set of tokens for the entity, wherein the entity is represented in a second modality by the second representation;
embed the second set of tokens into a second numeric matrix representation;
generate a second positional encoding for the second set of tokens, wherein the second positional encoding preserves a second order of the tokens included in the second set of tokens;
provide, to the second encoder, the second numeric matrix representation and the second positional encoding;
cause the second encoder to encode the second numeric matrix representation and the second positional encoding, such that the second encoder is configured to output a second set of embeddings; and
configure a projection layer of the machine learning model to project the first set of embeddings and the second set of embeddings to a shared contrastive space.

### Claim 2 (depends on 1)

The computing system of claim 1, the one or more computer-readable instructions being further executable to further configure the computing system to:
configure a global average pooling layer of the machine learning model to integrate the first set of embeddings and second set of embeddings prior to the projection layer projecting the first set of embeddings and second set of embeddings to the shared contrastive space.

### Claim 3

A method comprising:
accessing a first encoder of a machine learning model;
tokenizing a first representation of an entity, resulting in generation of a first set of tokens for the entity, wherein the entity is represented in a first modality by the first representation;
embedding the first set of tokens into a first numeric matrix representation;
generating a first positional encoding for the first set of tokens, wherein the first positional encoding preserves a first order of the tokens included in the first set of tokens;
providing, to the first encoder, the first numeric matrix representation and the first positional encoding;
causing the first encoder to encode the first numeric matrix representation and the first positional encoding, resulting in generation of a first set of encoded data;
accessing a second encoder of the machine learning model;
tokenizing a second representation of the entity, resulting in generation of a second set of tokens for the entity, wherein the entity is represented in a second modality by the second representation;
embedding the second set of tokens into a second numeric matrix representation;
generating a second positional encoding for the second set of tokens, wherein the second positional encoding preserves a second order of the tokens included in the second set of tokens;
providing, to the second encoder, the second numeric matrix representation and the second positional encoding;
causing the second encoder to encode the second numeric matrix representation and the second positional encoding, resulting in generation of a second set of encoded data; and
configuring a projection layer of the machine learning model to project the first set of encoded data and the second set of encoded data to a shared contrastive space.

### Claim 4 (depends on 3)

The method of claim 3, wherein the entity is related to a domain.

### Claim 5 (depends on 4)

The method of claim 4, wherein the domain is a chemical domain.

### Claim 6 (depends on 4)

The method of claim 4, wherein the domain is an enterprise domain.

### Claim 7 (depends on 3)

The method of claim 3, wherein the first modality is an image modality.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second modality is an audio modality.

### Claim 9 (depends on 3)

The method of claim 3, wherein the first modality is a text modality or an audio modality.

### Claim 10 (depends on 9)

The method of claim 9, wherein the second modality is a different one of the text modality or the audio modality.

### Claim 11 (depends on 3)

The method of claim 3, wherein at least one of the first or second encoders is configured as a plurality of transformer encoder blocks.

### Claim 12

A computer system comprising:
one or more processors; and
one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
access a first encoder of a machine learning model;
tokenize a first representation of an entity, resulting in generation of a first set of tokens for the entity, wherein the entity is represented in a first modality by the first representation;
embed the first set of tokens into a first numeric matrix representation;
generate a first positional encoding for the first set of tokens, wherein the first positional encoding preserves a first order of the tokens included in the first set of tokens;
provide, to the first encoder, the first numeric matrix representation and the first positional encoding;
cause the first encoder to encode the first numeric matrix representation and the first positional encoding, resulting in generation of a first set of encoded data;
access a second encoder of the machine learning model;
tokenize a second representation of the entity, resulting in generation of a second set of tokens for the entity, wherein the entity is represented in a second modality by the second representation;
embed the second set of tokens into a second numeric matrix representation;
generate a second positional encoding for the second set of tokens, wherein the second positional encoding preserves a second order of the tokens included in the second set of tokens;
provide, to the second encoder, the second numeric matrix representation and the second positional encoding;
cause the second encoder to encode the second numeric matrix representation and the second positional encoding, resulting in generation of a second set of encoded data; and
project the first set of encoded data and the second set of encoded data to a shared contrastive space.

### Claim 13 (depends on 12)

The computer system of claim 12, wherein the entity is related to a domain.

### Claim 14 (depends on 13)

The computer system of claim 13, wherein the domain is a chemical domain.

### Claim 15 (depends on 13)

The computer system of claim 13, wherein the domain is an enterprise domain.

### Claim 16 (depends on 12)

The computer system of claim 12, wherein the first modality is an image modality.

### Claim 17 (depends on 16)

The computer system of claim 16, wherein the second modality is an audio modality.

### Claim 18 (depends on 12)

The computer system of claim 12, wherein the first modality is one of an image modality, a text modality, or an audio modality.

### Claim 19 (depends on 18)

The computer system of claim 18, wherein the second modality is a different one of the image modality, the text modality, or the audio modality.

### Claim 20 (depends on 12)

The computer system of claim 12, wherein at least one of the first or second encoders is configured as a plurality of transformer encoder blocks.