Multimodal domain embeddings via contrastive learning

A method comprising:

• accessing a first encoder of a machine learning model;
• tokenizing a first representation of an entity, resulting in generation of a first set of tokens for the entity, wherein the entity is represented in a first modality by the first representation;
• embedding the first set of tokens into a first numeric matrix representation;
• generating a first positional encoding for the first set of tokens, wherein the first positional encoding preserves a first order of the tokens included in the first set of tokens;
• providing, to the first encoder, the first numeric matrix representation and the first positional encoding;
• causing the first encoder to encode the first numeric matrix representation and the first positional encoding, resulting in generation of a first set of encoded data;
• accessing a second encoder of the machine learning model;
• tokenizing a second representation of the entity, resulting in generation of a second set of tokens for the entity, wherein the entity is represented in a second modality by the second representation;
• embedding the second set of tokens into a second numeric matrix representation;
• generating a second positional encoding for the second set of tokens, wherein the second positional encoding preserves a second order of the tokens included in the second set of tokens;
• providing, to the second encoder, the second numeric matrix representation and the second positional encoding;
• causing the second encoder to encode the second numeric matrix representation and the second positional encoding, resulting in generation of a second set of encoded data;
• and
• configuring a projection layer of the machine learning model to project the first set of encoded data and the second set of encoded data to a shared contrastive space.

A computer system comprising:

• one or more processors;
• and
• one or more hardware storage devices that store instructions that are executable by the one or more processors to cause the computer system to:
• access a first encoder of a machine learning model;
• tokenize a first representation of an entity, resulting in generation of a first set of tokens for the entity, wherein the entity is represented in a first modality by the first representation;
• embed the first set of tokens into a first numeric matrix representation;
• generate a first positional encoding for the first set of tokens, wherein the first positional encoding preserves a first order of the tokens included in the first set of tokens;
• provide, to the first encoder, the first numeric matrix representation and the first positional encoding;
• cause the first encoder to encode the first numeric matrix representation and the first positional encoding, resulting in generation of a first set of encoded data;
• access a second encoder of the machine learning model;
• tokenize a second representation of the entity, resulting in generation of a second set of tokens for the entity, wherein the entity is represented in a second modality by the second representation;
• embed the second set of tokens into a second numeric matrix representation;
• generate a second positional encoding for the second set of tokens, wherein the second positional encoding preserves a second order of the tokens included in the second set of tokens;
• provide, to the second encoder, the second numeric matrix representation and the second positional encoding;
• cause the second encoder to encode the second numeric matrix representation and the second positional encoding, resulting in generation of a second set of encoded data;
• and
• project the first set of encoded data and the second set of encoded data to a shared contrastive space.