US 11,600,087 B2Granted

Convolutional neural networks for locating objects of interest in images of biological samples

AssigneeVentana Medical Systems, Inc.

InventorsSrinivas Chukka, Jianxu Chen

Life of this patent

Estimating the term…

Loading patent…

Abstract

Convolutional neural networks for detecting objects of interest within images of biological specimens are disclosed. Also disclosed are systems and methods of training and using such networks, one method including: obtaining a sample image and at least one of a set of positive points and a set of negative points, wherein each positive point identifies a location of one object of interest within the sample image, and each negative point identifies a location of one object of no-interest within the sample image; obtaining one or more predefined characteristics of objects of interest and/or objects of no-interest, and based on the predefined characteristics, generating a boundary map comprising a positive area around each positive point the set of positive points, and/or a negative area around each negative point in the set of negative points; and training the convolutional neural network using the sample image and the boundary map.

Claims20

Dependent claims

A system configured to access and use a computer-implemented convolutional neural network to identify a plurality of objects of interest from an image depicting at least part of a biological sample, wherein the plurality of objects of interest correspond to a particular type of cell, and wherein the convolutional neural network comprises:

a plurality of encoders comprising a first encoder that comprises a first convolutional unit, a rectified linear unit, and a downsampling unit, the first encoder being configured to:
receive encoder input data that represents the image;
process the encoder input data with at least the first convolutional unit and the rectified linear unit to produce processed data;
downsample the processed data with the downsampling unit to produce downsampled processed data;
and
output the processed data and the downsampled processed data;
and
a plurality of decoders comprising a first decoder that comprises an upsampling unit and a second convolutional unit, the first decoder being configured to:
receive decoder input data representing the image from a second decoder from the plurality of decoders or from a second encoder from the plurality of encoders;
upsample the decoder input data to produce upsampled data;
receive the processed data from the first encoder;
combine the upsampled data with the processed data to produce combined upsampled data;
process the combined upsampled data with at least the second convolutional unit to produce processed combined upsampled data;
and
output the processed combined upsampled data.

12 more elements

A method comprising:

accessing an image depicting at least part of a biological sample;
and
applying a convolutional neural network to the image to generate an output identifying a plurality of objects of interest from the image, wherein the plurality of objects of interest correspond to a particular type of cell, and wherein applying the convolutional neural network includes:
receiving, by a first encoder of the convolutional neural network, encoder input data that represents the image, wherein the first encoder includes a first convolutional unit, a rectified linear unit, and a downsampling unit;
processing, by the first encoder, the encoder input data with at least a first convolutional unit and a rectified linear unit to produce processed data;
downsampling, by the first encoder, the processed data with a downsampling unit to produce downsampled processed data;
outputting, by the first encoder, the processed data and the downsampled processed data;
receiving, by a first decoder of the convolutional neural network, decoder input data representing the image from a second decoder from a plurality of decoders or from a second encoder from a plurality of encoders, wherein the first decoder includes an upsampling unit and a second convolutional unit;
upsampling, by the first decoder, the decoder input data to produce upsampled data;
receiving, by the first decoder, the processed data from the first encoder;
combining, by the first decoder, the upsampled data with the processed data to produce combined upsampled data;
processing, by the first decoder, the combined upsampled data with at least the second convolutional unit to produce processed combined upsampled data;
and
outputting, by the first decoder, the processed combined upsampled data.

11 more elements

A non-transitory computer-readable medium comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform one or more operations comprising:

accessing an image depicting at least part of a biological sample;
and
applying a convolutional neural network to the image to generate an output identifying a plurality of objects of interest from the image, wherein the plurality of objects of interest correspond to a particular type of cell, and wherein applying the convolutional neural network includes:
receiving, by a first encoder of the convolutional neural network, encoder input data that represents the image, wherein the first encoder includes a first convolutional unit, a rectified linear unit, and a downsampling unit;
processing, by the first encoder, the encoder input data with at least a first convolutional unit and a rectified linear unit to produce processed data;
downsampling, by the first encoder, the processed data with a downsampling unit to produce downsampled processed data;
outputting, by the first encoder, the processed data and the downsampled processed data;
receiving, by a first decoder of the convolutional neural network, decoder input data representing the image from a second decoder from a plurality of decoders or from a second encoder from a plurality of encoders, wherein the first decoder includes an upsampling unit and a second convolutional unit;
upsampling, by the first decoder, the decoder input data to produce upsampled data;
receiving, by the first decoder, the processed data from the first encoder;
combining, by the first decoder, the upsampled data with the processed data to produce combined upsampled data;
processing, by the first decoder, the combined upsampled data with at least the second convolutional unit to produce processed combined upsampled data;
and
outputting, by the first decoder, the processed combined upsampled data.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/459,125Filed 27 Aug 2021 · division
Priority: 16 Nov 2016
Published: US 2021/0390281 A116 Dec 2021
Granted: 7 Mar 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification16 codes

G06V 20/69
Type of objects: microscopic objects, e.g. biological cells or cellular parts
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…
G06F 18/24143
Distances to prototypes: distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
G06N 3/045
Architecture, e.g. interconnection topology: combinations of networks

Family

Application62/423,114Us provisional application · filed 16 Nov 2016
PCT/EP2017/079246Parent of this division · filed 15 Nov 2017
US 2021/0390281 A1Pre-grant publication · published 16 Dec 2021
Application16/612,323Parent of this division
US 11,132,529 B2Parent of this division
US 2020/0097701 A1Published application
US 11,600,087 B2This patent

From the US filing history. Foreign family members aren’t listed.