# US 12,118,715 B2 · Systems and methods for image classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,715 B2
- Document ID: US12118715B2
- Status: Granted
- Kind: B2
- Application number: 16/778,179
- Filed: 2020-01-31
- Published: 2020-08-06
- Granted: 2024-10-15
- Earliest claimed: 2019-02-01
- Assignees: Owkin, Inc.; Owkin France SAS
- Applicants: Owkin Inc.; Owkin France SAS
- Inventors: Pierre Courtiol; Eric W. Tramel; Marc Sanselme; Gilles Wainrib
- Web page: https://patentgrep.com/patent/US12118715B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 18/2163: {Partitioning the feature space}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G06T 2207/10024: Color image
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus of a device that classifies an image is described. In an exemplary embodiment, the method includes tiling at least one region of interest of the input image into a set of tiles. For each tile, the method includes extracting a feature vector of the tile by applying a convolutional neural network, wherein a feature is a local descriptor of the tile; and computing a score of the tile from the extracted feature vector, said tile score being representative of a contribution of the tile into a classification of the input image. The method also includes sorting a set of the tile scores and selecting a subset of the tile scores based on their value and/or their rank in the sorted set. The method also includes applying a classifier to the selected tile scores in order to classify the input image.

## Claims

### Claim 1

A method for classifying an input image comprising:
tiling at least one region of interest of the input image into a plurality of tiles; for each tile, extracting a feature vector of the tile by applying a convolutional neural network, wherein a feature is a local descriptor of the tile,
processing the extracted feature vectors of a subset of the plurality of tiles selected from the plurality of tiles to classify the input image using a plurality of tile scores of the plurality of tiles computed using the extracted feature vectors, wherein the subset of the plurality of tiles has a smaller number of tiles than the plurality of tiles; and
applying a classifier to the selected tile scores in order to classify the input image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the classifying comprises:
computing a score of the tile from the extracted feature vector, said tile score being representative of a contribution of the tile into a classification of the input image; and
sorting a set of the tile scores and selecting a subset of the tile scores based on their value and/or their rank in the sorted set.

### Claim 3 (depends on 2)

The method according to claim 2, wherein, for each tile, the score of the tile is computed by applying at least one one-dimensional convolutional layer to the extracted feature vector of the tile.

### Claim 4 (depends on 2)

The method according to claim 2, wherein said classifier is a multi-layer perceptron classifier.

### Claim 5 (depends on 4)

The method of claim 4, wherein the multi-layer perceptron classifier comprises two fully connected layers.

### Claim 6 (depends on 2)

The method according to claim 2, wherein the input image is a histopathology slide, the region of interest is a tissue region, and the classification of the image is a diagnosis classification.

### Claim 7 (depends on 2)

The method according to claim 2, wherein selecting a subset of the tile scores comprises:
selecting a first given number, Rtop, of highest values of the tile scores and a second given number, Rbottom, of smallest values of the tile scores.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the tiling comprises detecting the at least one region of interest using masks.

### Claim 9 (depends on 8)

The method of claim 8 further comprising generating, using Otsu's method applied to hue and saturation channels, after transformation of the input image into hue, saturation, value (HSV) color space.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the tiling comprises normalizing colors of the input image.

### Claim 11 (depends on 10)

The method of claim 10 wherein the normalizing comprises applying a left-shift to RGB channels of the input image and resealing between two given values.

### Claim 12 (depends on 1)

The method according to claim 1, wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the plurality of tiles has a predetermined size.

### Claim 13 (depends on 1)

The method according to claim 1, wherein the convolutional neural network is a ResNet-50 type of residual neural network with a last layer removed using a previous layer as output.

### Claim 14

A non-transitory computer readable medium with a memory storing code instructions which, when executed by a processor, cause the processor to perform operations for classifying an input image, the operations comprising:
tiling at least one region of interest of the input image into a plurality of tiles; for each tile, extracting a feature vector of the tile by applying a convolutional neural network, wherein a feature is a local descriptor of the tile;
processing the extracted feature vectors of a subset of tiles selected from the plurality of tiles to classify the input image using a plurality of tile scores of the plurality of tiles computed using the extracted feature vectors, wherein the subset of the plurality of tiles has a smaller number of tiles than the plurality of tiles; and
applying a classifier to the selected tile scores in order to classify the input image.

### Claim 15 (depends on 14)

The non-transitory computer readable medium of claim 14, wherein the classifying comprises:
computing a score of the tile from the extracted feature vector, said tile score being representative of a contribution of the tile into a classification of the input image; and
sorting a set of the tile scores and selecting a subset of the tile scores based on their value and/or their rank in the sorted set.

### Claim 16

A method for predicting a global score of an input image comprising:
tiling at least a region of interest of the input image into a plurality of tiles; for each tile, extracting a feature vector of the tile by applying a convolutional neural network, wherein a feature is a local descriptor of the tile; and
processing the extracted feature vectors of a subset of tiles selected from the plurality of tiles to obtain the global score of the input image by classifying the input image using a plurality of tile scores of the plurality of tiles computed from the extracted feature vectors, wherein the classification generates the global score for the input image and the subset of the plurality of tiles has a smaller number of tiles than the plurality of tiles.

### Claim 17 (depends on 16)

The method according to claim 16, wherein the classifying comprises:
computing a score of the tile from the extracted feature vector, said tile score being representative of a contribution of the tile into the global score of the input image;
sorting a set of the tile scores and selecting a subset of the tile scores based on their value and/or their rank in the sorted set; and
applying a regressor to the kept tile scores in order to obtain the global score of the input image.

### Claim 18 (depends on 17)

The method according to claim 17, wherein, for each tile, the score of the tile is computed by applying at least one one-dimensional convolutional layer to the extracted feature vector of the tile.

### Claim 19 (depends on 17)

The method according to claim 17, wherein said regressor is a multi-layer perceptron regressor.

### Claim 20 (depends on 19)

The method of claim 19, wherein the multi-layer perceptron regressor comprises two fully connected layers.

### Claim 21 (depends on 17)

The method according to claim 17, wherein the input image is a histopathology slide, the region of interest is a tissue region, and the global score is a risk score correlated with a prognosis, or correlated with a response to a treatment.

### Claim 22 (depends on 17)

The method according to claim 17, wherein selecting a subset of the tile scores comprises:
selecting a first given number, Rtop, of a highest value of the tile scores and a second given number, Rbottom, of a smallest value of the tile scores are selected at operation (c).

### Claim 23 (depends on 16)

The method according to claim 16, wherein the tiling comprises detecting said at least one region of interest using masks.

### Claim 24 (depends on 23)

The method of claim 23 further comprising generating, using Otsu's method applied to hue and saturation channels, after transformation of the input image into hue, saturation, value (HSV) color space.

### Claim 25 (depends on 16)

The method according to claim 16, wherein the tiling comprises normalizing colors of the input image.

### Claim 26 (depends on 25)

The method of claim 25, wherein the normalizing comprises applying a left-shift to RGB channels of the input image and rescaling it between two given values.

### Claim 27 (depends on 16)

The method according to claim 16, wherein the tiling comprises:
applying a fixed tiling grid to at least the region of interest, wherein each of the plurality of tiles has a predetermined size.

### Claim 28 (depends on 16)

The method according to claim 16, wherein the convolutional neural network is a ResNet-50 type of residual neural network with a last layer removed using a previous layer as output.

### Claim 29

A non-transitory computer readable medium with a memory storing code instructions which, when executed by a processor, cause the processor to perform operations for predicting a global score of an input image, the operations comprising:
tiling at least a region of interest of the input image into a plurality of tiles; for each tile, extracting a feature vector of the tile by applying a convolutional neural network, wherein a feature is a local descriptor of the tile; and
processing the extracted feature vectors of a subset of tiles selected from the plurality of tiles to obtain the global score of the input image by classifying the input image using a plurality of tile scores of the plurality of tiles computed from the extracted feature vectors, wherein the classification generates the global score for the input image and the subset of the plurality of tiles has a smaller number of tiles.

### Claim 30 (depends on 29)

The non-transitory computer readable medium of claim 29, wherein the classifying comprises:
computing a score of the tile from the extracted feature vector, said tile score being representative of a contribution of the tile into the global score of the input image;
sorting a set of the tile scores and selecting a subset of the tile scores based on their value and/or their rank in the sorted set; and
applying a regressor to the kept tile scores in order to obtain the global score of the input image.