# US 10,769,501 B1 · Analysis of perturbed subjects using semantic embeddings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,501 B1
- Document ID: US10769501B1
- Status: Granted
- Kind: B1
- Application number: 16/133,542
- Filed: 2018-09-17
- Granted: 2020-09-08
- Earliest claimed: 2017-02-15
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Dale M. Ando; Marc Berndl; Lusann Yang; Michelle Dimon
- Web page: https://patentgrep.com/patent/US10769501B1

## CPC

- G01N 33/5005: {involving human or animal cells (immunoassay G01N33/56966; immunoassays of protozoa G01N33/56905; protozoa in screening assays C12Q1/025)}
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24: Classification techniques
- G06F 18/25: Fusion techniques
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/091: Active learning
- G06N 3/096: Transfer learning
- G06V 10/82: using neural networks
- G06V 20/698: {Matching; Classification}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G01N 33/20: Metals
- G01N 33/48: Biological material, e.g. blood, urine (G01N33/02, G01N33/26, G01N33/44, G01N33/46 take precedence); Haemocytometers (counting blood corpuscules distributed over a surface by scanning the surface G06M11/02)
- G06V 2201/10: Recognition assisted with metadata
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure relates to analysis of perturbed subjects using semantic embeddings. One example embodiment includes a method. The method includes applying a respective perturbation to each of a plurality of subjects in a controlled environment. The method also includes producing a respective visual representation for each of the perturbed subjects using at least one imaging modality. Further, the method includes obtaining, by a computing device for each of the respective visual representations, a corresponding semantic embedding associated with the respective visual representation. The semantic embedding associated with the respective visual representation is generated using a machine-learned, deep metric network model. In addition, the method includes classifying, by the computing device based on the corresponding semantic embedding, each of the visual representations into one or more groups.

## Claims

### Claim 1

A method comprising:
applying a respective perturbation to each of a plurality of subjects in a controlled environment;
producing a respective visual representation for each of the perturbed subjects using at least one imaging modality;
obtaining, by a computing device for each of the respective visual representations, a corresponding semantic embedding associated with the respective visual representation,
wherein the semantic embedding associated with the respective visual representation is generated using a machine-learned, deep metric network model,
wherein obtaining the corresponding semantic embedding associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,
wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and
wherein incorporating data captured during one or more respective supplementary measurements comprises:
generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;
generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and
generating a composite semantic embedding:
(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or
(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and
classifying, by the computing device based on the corresponding semantic embedding, each of the visual representations into one or more groups.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one imaging modality comprises computed tomography, magnetic resonance imaging, positron emission tomography, ultrasound, x-ray computed tomography, x-ray diffraction, fluoroscopy, projectional radiography, single-photon emission computed tomography, scintigraphy, elastography, photoacoustic imaging, near-infrared spectroscopy, magnetic particle imaging, optoacoustic imaging, diffuse optical tomography, Raman spectroscopy, fluorescent microscopy, confocal microscopy, two-photon microscopy, hyperspectral analysis, transmission microscopy, electromagnetic scanning, differential interference contrast microscopy, multiphoton microscopy, dark-field microscopy, quantitative phase-contrast microscopy, near-field scanning optical microscopy, photo-activated localization microscopy, second harmonic imaging, holography, scanning electron microscopy, or tunneling electron microscopy.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of subjects comprises biological cells, fibroblasts, malaria cells, yeast cells, yeast cultures, bacteria, bacterial cultures, fungus, fungal cultures, cancer cells, blood cells, malarial parasites, mitochondria, nuclei, axons, dendrites, induced pluripotent stem cells, biological cells from a given region of an organism, biological cells from a given tissue of an organism, biological cells from a given organ of an organism, biological cells from a given system of an organism, biological cell ensembles, tissues, organs, organoids, biological systems, organisms, groups of organisms, ecosystems, chemical compounds, crystals, metallic glasses, mixtures of metallic salts, semiconductors, metals, dielectrics, graphene, microelectromechanical systems (MEMS), or nanoelectromechanical systems (NEMS).

### Claim 4 (depends on 1)

The method of claim 1, wherein applying the respective perturbation to each of the plurality of subjects comprises at least one of:
adding one or more small molecule compounds to at least one of the plurality of subjects;
applying a candidate treatment compound to at least one of the plurality of subjects;
applying a genetic modification to at least one of the plurality of subjects;
allowing a predetermined amount of time to elapse for at least one of the plurality of subjects;
illuminating at least one of the plurality of subjects with light of a predetermined wavelength;
heating or cooling at least one of the plurality of subjects to a predetermined temperature;
exposing at least one of the plurality of subjects to another subject of the plurality of subjects;
introducing at least one of the plurality of subjects to a different environment; or
applying a predetermined force to one or more regions of at least one of the plurality of subjects.

### Claim 5 (depends on 1)

The method of claim 1, wherein each of the respective visual representations comprises a three-dimensional visual representation.

### Claim 6 (depends on 5)

The method of claim 5, wherein the machine-learned, deep metric network model was trained using three-dimensional training data.

### Claim 7 (depends on 1)

The method of claim 1, wherein each of the respective visual representations comprises a videographic representation.

### Claim 8 (depends on 7)

The method of claim 7, wherein the machine-learned, deep metric network model was trained using videographic training data.

### Claim 9 (depends on 1)

The method of claim 1, wherein producing the respective visual representations comprises performing a patch selection to identify each of the respective perturbed subjects.

### Claim 10 (depends on 9)

The method of claim 9, wherein the patch selection is machine-learned and comprises an attention mechanism used to define which regions of a preliminary visual representation are of threshold interest level to be selected by the patch selection.

### Claim 11 (depends on 1)

The method of claim 1, further comprising, for each of the semantic embeddings:
detecting, by the computing device, a background effect present within the semantic embedding; and
removing, by the computing device, the background effect from the semantic embedding.

### Claim 12 (depends on 11)

The method of claim 11, wherein the data captured during one or more respective supplementary measurements comprises experimental metadata, transcriptomic data, genomic data, proteomic data, metabolomic data, lipidomic data, bulk semiconductor material properties, or patient diagnostic data.

### Claim 13 (depends on 11)

The method of claim 11,
wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type,
wherein each visual representation comprises a plurality of channels, and
wherein incorporating data captured during one or more respective supplementary measurements comprises:
retrieving, by the computing device, each channel of the respective visual representation;
generating, by the computing device using the machine-learned, deep metric network model, a respective single-channel semantic embedding for each channel of the respective visual representation;
generating, by the computing device using the machine-learned, deep metric network model, an additional single-channel semantic embedding for the data captured during one or more respective supplementary measurements; and
concatenating, by the computing device, the single-channel semantic embeddings for each channel of the respective visual representation with the additional single-channel semantic embedding into a multi-channel semantic embedding.

### Claim 14 (depends on 1)

The method of claim 1,
wherein the composite semantic embedding is generated (i) using the additional machine-learned model, and wherein the composite semantic embedding comprises the hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding.

### Claim 15 (depends on 1)

The method of claim 1,
wherein the composite semantic embedding is generated (ii) by applying the mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding.

### Claim 16 (depends on 1)

The method of claim 1, wherein classifying each of the visual representations into one or more groups comprises performing vector arithmetic on the corresponding semantic embedding to determine to which of the one or more groups the respective visual representation belongs.

### Claim 17 (depends on 1)

The method of claim 1,
wherein classifying each of the visual representations into one or more groups comprises applying a classification model to each of the corresponding semantic embeddings, and
wherein the classification model is trained to identify responses to the respective perturbations and disregard statistical variations present across the plurality of subjects.

### Claim 18 (depends on 17)

The method of claim 17, wherein the classification model was trained using a human-in-the-loop procedure to identify which variations correspond to notable differences and which variations are attributable to randomness.

### Claim 19 (depends on 1)

The method of claim 1, wherein classifying the visual representations into one or more groups comprises:
identifying one or more values corresponding to one or more respective dimensions of the semantic embedding associated with the respective visual representation; and
determining, based on the one or more values, to which of an enumerated list of groups the respective visual representation belongs.

### Claim 20 (depends on 1)

The method of claim 1,
wherein classifying the visual representations into one or more groups comprises extracting information about the respective visual representation based on a topology within the semantic embedding associated with the respective visual representation, and
wherein the topology was enforced during training of the machine-learned, deep metric network model.

### Claim 21 (depends on 20)

The method of claim 20, wherein the topology is linear, elliptical, circular, polygonal, spherical, ellipsoidal, cylindrical, conical, hyperboloidal, paraboloidal, toroidal, pyramidal, polyhedral, or defined in a dimensional space having greater than three dimensions.

### Claim 22 (depends on 1)

The method of claim 1,
wherein the machine-learned, deep metric network model is trained using a plurality of photographic images as training data,
wherein the photographic images are query results ranked based on selections, and
wherein rankings of the photographic images are used in the training of the machine-learned, deep metric network model to determine image similarity between two images.

### Claim 23 (depends on 1)

The method of claim 1, further comprising training the machine-learned, deep metric network model, wherein training the machine-learned, deep metric network model comprises:
receiving, by the computing device, a series of three-image sets as training data,
wherein each three-image set comprises a query image, a positive image, and a negative image,
wherein the query image, the positive image, and the negative image are query results ranked in comparison with one another based on selections, and
wherein the selections indicate that a similarity between the query image and the positive image is greater than a similarity between the query image and the negative image; and
refining, by the computing device, the machine-learned, deep metric network model based on each three-image set to account for image components of the query image, the positive image, and the negative image.

### Claim 24 (depends on 1)

The method of claim 1, further comprising determining, by the computing device, a preferred perturbation from among the respective perturbations based on the one or more groups into which each of the visual representations are classified.

### Claim 25

A non-transitory, computer-readable medium having instructions stored thereon, wherein the instructions, when executed by a processor, cause the processor to execute a method comprising:
for each respective subject of a plurality of subjects, receiving a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;
obtaining, for each of the visual representations, a corresponding semantic embedding associated with the visual representation,
wherein the semantic embeddings are generated using a machine-learned, deep metric network model,
wherein obtaining the semantic embeddings associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,
wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and
wherein incorporating data captured during one or more respective supplementary measurements comprises:
generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;
generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and
generating a composite semantic embedding:
(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or
(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and
classifying, based on the corresponding semantic embedding, each of the visual representations into one or more groups.

### Claim 26

A method comprising:
for each respective subject of a plurality of subjects, receiving, by a computing device, a corresponding visual representation of the respective subject that was produced using at least one imaging modality after application of a respective perturbation to the respective subject in a controlled environment;
obtaining, by the computing device, a semantic embedding associated with the visual representation of the perturbed subject,
wherein the semantic embedding associated with the visual representation of the perturbed subject is generated using a machine-learned, deep metric network model,
wherein obtaining the corresponding semantic embedding associated with the respective visual representations comprises incorporating data captured during one or more respective supplementary measurements of a given measurement type,
wherein the machine-learned, deep metric network model was trained using training data that includes data of the given measurement type, and
wherein incorporating data captured during one or more respective supplementary measurements comprises:
generating a first preliminary semantic embedding for the respective visual representation using the machine-learned, deep metric network model;
generating a second preliminary semantic embedding for the data captured during one or more respective supplementary measurements using the machine-learned, deep metric network model; and
generating a composite semantic embedding:
(i) using an additional machine-learned model, wherein the composite semantic embedding comprises a hybrid of the first preliminary semantic embedding and the second preliminary semantic embedding; or
(ii) by applying a mathematical operation to corresponding dimensions of the first preliminary semantic embedding and the second preliminary semantic embedding; and
determining, by the computing device based on the semantic embedding, what type of perturbation was applied to the perturbed subject.