Analysis of perturbed subjects using semantic embeddings

A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method comprising:

• for each respective subject of a plurality of subjects, receiving a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;
• obtaining, for each of the visual representations, a corresponding semantic embedding associated with the visual representation,
• wherein the semantic embeddings are generated using a machine-learned, deep metric network model,
• wherein obtaining the semantic embeddings associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,
• wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and
• wherein incorporating data captured during one or more respective supplementary measurements comprises:
• generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;
• generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model;
• and
• generating a composite semantic embedding:
• (i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding;
• or
• (ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding;
• and
• classifying, based on the corresponding semantic embedding, each of the visual representations into one or more groups.

A method comprising:

• for each respective subject of a plurality of subjects, receiving, by a computing device, a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;
• obtaining, by the computing device, a semantic embedding associated with the visual representation of the perturbed subject,
• wherein the semantic embedding associated with the visual representation of the perturbed subject is generated using a machine-learned, deep metric network model,
• wherein obtaining the corresponding semantic embedding associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,
• wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and
• wherein incorporating data captured during one or more respective supplementary measurements comprises:
• generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;
• generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model;
• and
• generating a composite semantic embedding:
• (i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding;
• or
• (ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding;
• and
• determining, by the computing device based on the semantic embedding, what type of perturbation was applied to the perturbed subject.