# US 2020/0394459 A1 · Cell image synthesis using one or more neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394459 A1
- Document ID: US20200394459A1
- Status: Published application
- Kind: A1
- Application number: 16/443,549
- Filed: 2019-06-17
- Published: 2020-12-17
- Earliest claimed: 2019-06-17
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Ziyue Xu; Xiaosong Wang; Hoo Chang Shin; Dong Yang; Holger Roth; Daguang Xu; Ling Zhang; Fausto Milletari
- Web page: https://patentgrep.com/patent/US20200394459A1

## CPC

- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 5/04: Inference or reasoning models
- G06T 7/11: Region-based segmentation
- G16B 25/00: ICT specially adapted for hybridisation; ICT specially adapted for gene or protein expression
- G16B 40/00: ICT specially adapted for biostatistics; ICT specially adapted for bioinformatics-related machine learning or data mining, e.g. knowledge discovery or pattern finding
- G16B 45/00: ICT specially adapted for bioinformatics-related data visualisation, e.g. displaying of maps or networks
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30061: Lung
- G06T 2210/41: Medical
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY

## Abstract

Apparatuses, systems, and techniques to generate synthesized images including digital representations of groups of cells blended realistically with appropriate background images. In at least one embodiment, background image data and gene expression data are fused together to generate such a synthesized image using one or more neural networks.

## Claims

### Claim 1

A processor, comprising:
one or more arithmetic logic units (ALUs) to help generate one or more images of one or more cells based, at least in part, on genetic information associated with the one or more cells.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more ALUs are further to be configured to:
accept, as input, background image data and genetic expression data, the genetic expression data associated with visual features of the one or more cells.

### Claim 3 (depends on 1)

The processor of claim 1, wherein the one or more ALUs are further to be configured to:
infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using medical image data and genetic expression data.

### Claim 4 (depends on 3)

The processor of claim 3, wherein the GAN was trained in part by encoding the medical image data and the genetic expression data and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 5 (depends on 4)

The processor of claim 4, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic expression data to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values.

### Claim 6 (depends on 3)

The processor of claim 3, wherein the GAN as trained utilizes a learned genomic map between visual features of the one or more cells and the genetic expression data.

### Claim 7

A system comprising:
one or more memories to store genetic information associated with one or more cells; and
one or more processors to help generate one or more images of the one or more cells based, at least in part, on the genetic information.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more processors are further to be configured to:
accept, as input, background image data and genetic expression data, the genetic expression data associated with visual features of the one or more cells.

### Claim 9 (depends on 7)

The system of claim 7, wherein the one or more processors are further to be configured to:
infer the one or more images using a multi-conditional generative adversarial network (GAN) trained using medical image data and genetic expression data.

### Claim 10 (depends on 9)

The system of claim 9, wherein the GAN was trained in part by encoding the medical image data and the genetic expression data and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 11 (depends on 10)

The system of claim 10, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic expression data to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values.

### Claim 12 (depends on 9)

The system of claim 9, wherein the GAN as trained utilizes a learned genomic map between visual features of the one or more cells and the genetic expression data.

### Claim 13

A method comprising:
generating one or more images of one or more cells based, at least in part, on genetic information associated with the one or more cells; and
storing the one or more images.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
accepting, as input, background image data and genetic expression data, the genetic expression data associated with visual features of the one or more cells.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
inferring the one or more images using a multi-conditional generative adversarial network (GAN) trained using medical image data and genetic expression data.

### Claim 16 (depends on 15)

The method of claim 15, wherein the GAN was trained in part by encoding the medical image data and the genetic expression data and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 17 (depends on 16)

The method of claim 16, wherein the GAN was further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic expression data to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN were updated using the set of loss values.

### Claim 18 (depends on 15)

The method of claim 15, wherein the GAN as trained utilizes a learned genomic map between visual features of the one or more cells and the genetic expression data.

### Claim 19

A processor comprising:
one or more arithmetic logic units (ALUs) to help train one or more neural networks to be used to infer one or more images of one or more cells based, at least in part, on genetic information associated with the one or more cells.

### Claim 20 (depends on 19)

The processor of claim 19, wherein the one or more ALUs are further to be configured to:
accept, as training data, background image data and the genetic information associated with visual features of the one or more cells.

### Claim 21 (depends on 19)

The processor of claim 19, wherein the one or more neural networks include a multi-conditional generative adversarial network (GAN).

### Claim 22 (depends on 21)

The processor of claim 21, wherein the GAN is to be trained in part by encoding the medical image data and the genetic information and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 23 (depends on 22)

The processor of claim 22, wherein the GAN is to be further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic information to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN are to be updated using the set of loss values.

### Claim 24 (depends on 21)

The processor of claim 21, wherein the GAN is further trained to learn a genomic map between visual features of the one or more cells and the genetic information.

### Claim 25

A system, comprising:
one or more memories to store genetic information associated with one or more cells; and
one or more processors to train one or more neural networks to help infer one or more images of the one or more cells based, at least in part, on the genetic information.

### Claim 26 (depends on 25)

The system of claim 25, wherein the one or more processors are further to be configured to:
accept, as training data, background image data and the genetic information associated with visual features of the one or more cells.

### Claim 27 (depends on 25)

The system of claim 25, wherein the one or more neural networks include a multi-conditional generative adversarial network (GAN).

### Claim 28 (depends on 27)

The system of claim 27, wherein the GAN is to be trained in part by encoding the medical image data and the genetic information and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 29 (depends on 28)

The system of claim 28, wherein the GAN is to be further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic information to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN are to be updated using the set of loss values.

### Claim 30 (depends on 27)

The system of claim 27, wherein the GAN is further trained to learn a genomic map between visual features of the one or more cells and the genetic information.

### Claim 31

A method comprising:
training one or more neural networks to infer one or more images of one or more cells based, at least in part, on genetic information associated with the one or more cells; and
storing the neural network.

### Claim 32 (depends on 31)

The method of claim 31, further comprising:
accepting, as training data, background image data and the genetic information associated with visual features of the one or more cells.

### Claim 33 (depends on 31)

The method of claim 31, wherein the one or more neural networks include a multi-conditional generative adversarial network (GAN).

### Claim 34 (depends on 33)

The method of claim 33, wherein the GAN is to be trained in part by encoding the medical image data and the genetic information and fusing the encoded data to generate a synthetic image and a segmentation mask, the synthetic image including a representation of a group of cells blended with a background portion of the medical image data.

### Claim 35 (depends on 34)

The method of claim 34, wherein the GAN is to be further trained by passing the synthetic image, the segmentation mask, and a gene code for the genetic information to a discriminator for determining a set of loss values, wherein one or more network parameters of the GAN are to be updated using the set of loss values.

### Claim 36 (depends on 33)

The method of claim 33, wherein the GAN is further trained to learn a genomic map between visual features of the one or more cells and the genetic information.
