# US 12,118,721 B2 · Systems and methods for image preprocessing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,721 B2
- Document ID: US12118721B2
- Status: Granted
- Kind: B2
- Application number: 18/100,543
- Filed: 2023-01-23
- Published: 2023-11-30
- Granted: 2024-10-15
- Earliest claimed: 2019-06-25
- Assignees: Owkin, Inc.; Owkin France SAS
- Applicants: Owkin Inc.; Owkin France SAS
- Inventors: Pierre Courtiol; Olivier Moindrot; Charles Maussion; Charlie Saillard; Benoit Schmauch; Gilles Wainrib
- Web page: https://patentgrep.com/patent/US12118721B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2163: {Partitioning the feature space}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/23: Clustering techniques
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06V 10/32: Normalisation of the pattern dimensions
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G06T 2207/10056: Microscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method and apparatus of a device that classifies an image is described. In an exemplary embodiment, the device segments the image into a region of interest that includes information useful for classification and a background region by applying a first convolutional neural network. In addition, the device tiles the region of interest into a set of tiles. For each tile, the device extracts a feature vector of that tile by applying a second convolutional neural network, where the features of the feature vectors represent local descriptors of the tile. Furthermore, the device processes the extracted feature vectors of the set of tiles to classify the image.

## Claims

### Claim 1

A method of classifying an input image, the method comprising:
tiling a region of interest of the input image into a set of tiles;
extracting a first feature vector for the set of tiles by applying a first convolutional neural network, wherein the features of the feature vectors represent local descriptors of the region of interest;
extracting a second feature vector from the region of interest using a set of local labels generated for the input image;
combining the first and second feature vectors; and
processing the combined feature vectors to classify the input image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating the set of local labels by applying a second convolution neural network to the set of tiles.

### Claim 3 (depends on 2)

The method of claim 2, wherein the second convolution neural network is a one-dimensional convolution neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the classification is a global label for the digital image.

### Claim 5 (depends on 1)

The method of claim 1, wherein the extracting a first feature vector comprises:
extracting a feature vector for each of the tiles in the set of tiles.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
generating a local label for each of the tiles in the set of tiles.

### Claim 7 (depends on 1)

The method of claim 1, wherein the combining of the first and second feature vectors is a weighted pooling of the first feature vector and the second feature vector.

### Claim 8 (depends on 1)

The method of claim 1, wherein the combining of the first and second feature vectors is a concatenation of the first feature vector and the second feature vector.

### Claim 9 (depends on 1)

The method of claim 1, wherein the input image is a histopathology slide and the classification of the image is a diagnosis classification.

### Claim 10 (depends on 1)

The method of claim 1, wherein a classifier is used to classify the input image and the classifier is a multi-layer perceptron classifier, in particular comprising two fully connected layers.

### Claim 11

A non-transitory computer readable medium with a memory storing code instructions which, when executed by a processor, cause the processor to perform operations for classifying an input image, the operations comprising:
tiling a region of interest of the input image into a set of tiles;
extracting a first feature vector for the set of tiles by applying a first convolutional neural network, wherein the features of the feature vectors represent local descriptors of the region of interest;
extracting a second feature vector from the region of interest using a set of local labels generated for the input image;
combining the first and second feature vectors; and
processing the combined feature vectors to classify the input image.

### Claim 12 (depends on 11)

The non-transitory computer readable medium of claim 11, further comprising:
generating the set of local labels by applying a second convolution neural network to the set of tiles.

### Claim 13 (depends on 12)

The non-transitory computer readable medium of claim 12, wherein the second convolution neural network is a one-dimensional convolution neural network.

### Claim 14 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the classification is a global label for the digital image.

### Claim 15 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the extracting a first feature vector comprises:
extracting a feature vector for each of the tiles in the set of tiles.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, further comprising:
generating a local label for each of the tiles in the set of tiles.

### Claim 17 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the combining of the first and second feature vectors is a weighted pooling of the first feature vector and the second feature vector.

### Claim 18 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the combining of the first and second feature vectors is a concatenation of the first feature vector and the second feature vector.

### Claim 19 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein the input image is a histopathology slide and the classification of the image is a diagnosis classification.

### Claim 20 (depends on 11)

The non-transitory computer readable medium of claim 11, wherein a classifier is used to classify the input image and the classifier is a multi-layer perceptron classifier, in particular comprising two fully connected layers.