# US 12,008,764 B2 · Systems, devices, and methods for image processing to generate an image having predictive tagging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,764 B2
- Document ID: US12008764B2
- Status: Granted
- Kind: B2
- Application number: 18/170,076
- Filed: 2023-02-16
- Published: 2023-09-07
- Granted: 2024-06-11
- Earliest claimed: 2017-08-09
- Assignees: Allen Institute
- Applicants: Allen Institute
- Inventors: Gregory Johnson; Chawin Ounkomol; Forrest Collman; Sharmishtaa Seshamani
- Web page: https://patentgrep.com/patent/US12008764B2

## CPC

- G06T 7/11: Region-based segmentation
- A61B 5/0275: using tracers, e.g. dye dilution
- G01N 15/1429: Signal processing
- G01N 15/1433: using image recognition
- G01N 15/1468: {with spatial resolution of the texture or inner structure of the particle}
- G02B 21/008: {Details of detection or image processing, including general computer control}
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G06T 7/174: involving the use of two or more images
- G06T 7/187: involving region growing; involving region merging; involving connected component labelling
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G01N 2015/1006: {for cytology}
- G01N 2015/1488: {Methods for deciding}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10056: Microscopic image
- G06T 2207/10061: from scanning electron microscope
- G06T 2207/10064: Fluorescence image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computing device, method, system, and instructions in a non-transitory computer-readable medium for performing image analysis on 3D microscopy images to predict localization and/or labeling of various structures or objects of interest, by predicting the location in such images at which a dye or other marker associated with such structures would appear. The computing device, method, and system receives sets of 3D images that include unlabeled images, such as transmitted light images or electron microscope images, and labeled images, such as images captured with fluorescence tagging. The computing device trains a statistical model to associate structures in the labeled images with the same structures in the unlabeled light images. The processor further applies the statistical model to a new unlabeled image to generate a predictive labeled image that predicts the location of a structure of interest in the new image.

## Claims

### Claim 1

A computing device, comprising:
a communication interface configured to receive microscopy images;
a processor; and
a non-transitory computer-readable medium communicatively coupled to the processor and storing computer-executable instructions that, when executed by the processor, causes the processor to:
receive, via the communication interface, a first set of three-dimensional (3D) microscopy images and a second set of 3D microscopy images, wherein the first set of 3D microscopy images are 3D confocal laser scanning microscopy (CLSM) fluorescence images of a plurality of tissue samples each having a plurality of cells, and wherein the second set of 3D microscopy images are 3D transmitted light images of the same plurality of tissue samples, wherein fluorescence labeling is applied to the plurality of cells in the first set of 3D microscopy images, and wherein no fluorescence labeling is included in the second set of 3D microscopy images;
generate a neural network configured to convert a first type of image that is a 3D transmitted light image of cells to a second type of image that is a predicted 3D CLSM fluorescence image of the cells, wherein no fluorescence labeling is included in the first type of image, and wherein the instructions cause the processor to generate the neural network by training the neural network based on the first set of 3D microscopy images and the second set of 3D microscopy images;
receive, after the neural network is generated and trained, an additional 3D microscopy image that is a transmitted light image of an additional tissue sample having a plurality of cells, wherein no fluorescence labeling is included in the additional 3D microscopy image; and
generate, with the neural network and the additional 3D microscopy image, a predicted 3D CLSM fluorescence image that includes predicted fluorescence labeling of the plurality of cells for the additional tissue sample.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein the instructions further cause the processor to determine, using the predicted 3D CLSM fluorescence image, a cell characteristic of the plurality of cells of the additional tissue sample, wherein the cell characteristic is at least one of an average or median cell size, a cell count, cell morphology of at least one of the plurality of cells, a cell cycle phase of at least one of the plurality of cells, or the presence or absence a protein biomarker on a surface of at least one of the plurality of cells.

### Claim 3 (depends on 2)

The computing device of claim 2, wherein the neural network is a first neural network, wherein the instructions further cause the processor to:
receive an indication of which cell in the plurality of tissue samples have a classification of being a diseased cell,
generate a second neural network configured to convert the second type of image that is the predicted 3D CLSM fluorescence image to a predicted classification of whether the predicted fluorescence 3D CLSM image includes a diseased cell, wherein the instructions cause the processor to generate the second neural network by training the second neural network with predicted 3D CLSM fluorescence images generated by the first neural network and with the received indication of which cell in the plurality of tissue samples is a diseased cell; and
generate, with the second neural network and the predicted 3D CLSM fluorescence image of the additional tissue sample, a predicted classification of whether the additional tissue samples include a diseased cell.