# US 2021/0280311 A1 · Machine-learned hormone status prediction from image analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0280311 A1
- Document ID: US20210280311A1
- Status: Published application
- Kind: A1
- Application number: 16/895,983
- Filed: 2020-06-08
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: salesforce.com, inc.
- Applicants: salesforce.com, inc.
- Inventors: Nikhil Naik; Ali Madani; Nitish Shirish Keskar
- Web page: https://patentgrep.com/patent/US20210280311A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06V 20/69: Microscopic objects, e.g. biological cells or cellular parts
- G16H 10/20: for electronic clinical trials or questionnaires
- G16H 10/40: for data related to laboratory analysis, e.g. patient specimen analysis
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/10056: Microscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30068: Mammography; Breast
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

An analytics system uses one or more machine-learned models to predict a hormone receptor status from a H&E stain image. The system partitions H&E stain images each into a plurality of image tiles. Bags of tiles are created through sampling of the image tiles. The analytics system trains one or more machine-learned models with training H&E stain images having a positive or negative receptor status. The analytics system generates, via a tile featurization model, a tile feature vector for each image tile a test bag for a test H&E stain image. The analytics system generates, via an attention model, an aggregate feature vector for the test bag by aggregating the tile feature vectors of the test bag, wherein an attention weight is determined for each tile feature vector. The analytics system predicts a hormone receptor status by applying a prediction model to the aggregate feature vector for the test bag.

## Claims

### Claim 1

A method for predicting hormone receptor status using machine learning comprising:
accessing a test haematoxylin and eosin (H&E) stain image of a test tissue sample;
partitioning the test H&E stain image into a plurality of image tiles;
sampling a subset of image tiles from the plurality of image tiles of the test H&E stain image;
generating a tile feature vector for each image tile of the sampled subset of image tiles;
generating an aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the image tiles of the sampled subset; and
predicting a hormone receptor status by applying a machine-learned prediction model to the aggregate feature vector for the test H&E stain image, wherein the machine-learned prediction model is trained using a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status.

### Claim 2 (depends on 1)

The method of claim 1, wherein the hormone receptor status comprises an estrogen receptor status.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of image tiles are non-overlapping.

### Claim 4 (depends on 1)

The method of claim 1, further comprising filtering the plurality of image tiles to retain image tiles pertaining to the test tissue sample, wherein the sampled subset of image tiles is sampled from the image tiles pertaining to the test tissue sample.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the tile feature vector for each image tile comprises applying a tile featurization model to each image tile.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating the aggregate feature vector comprises applying an attention model to the tile feature vectors of the sampled subset of image tiles, wherein the attention model is configured to determine, for each tile feature vector of the sampled subset, an attention weight based on each tile feature vector and to sum products of each attention weight and the respective tile feature vector to produce the aggregate feature vector.

### Claim 7 (depends on 6)

The method of claim 6, wherein the featurization model, the attention model, and the machine-learned prediction model are trained synchronously using the first set of H&E stain images from the first set of tissue samples having the positive hormone receptor status and the second set of H&E stain images from the second set of tissue samples having the negative hormone receptor status.

### Claim 8 (depends on 1)

The method of claim 1, wherein the predicted hormone receptor status is one of:
a positive status or a negative status; and
a likelihood of a positive status or a likelihood of a negative status.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
sampling a second subset of image tiles from the plurality of image tiles of the test H&E stain image;
generating a tile feature vector for each image tile of the second sampled subset of image tiles;
generating a second aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the image tiles of the second sampled subset; and
predicting a second hormone receptor status by applying the machine-learned prediction model to the second aggregate feature vector, wherein a cumulative prediction of the hormone receptor status combines the hormone receptor status and the second hormone receptor status.

### Claim 10

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:
accessing a test haematoxylin and eosin (H&E) stain image of a test tissue sample;
partitioning the test H&E stain image into a plurality of image tiles;
sampling a subset of image tiles from the plurality of image tiles for the test H&E stain image;
generating a tile feature vector for each image tile of the sampled subset of image tiles;
generating an aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the image tiles in the sampled subset; and
predicting a hormone receptor status by applying a machine-learned prediction model to the aggregate feature vector for the test H&E stain image, wherein the machine-learned prediction model is trained using a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status.

### Claim 11 (depends on 10)

The storage medium of claim 10, the operations further comprising filtering the plurality of image tiles to retain image tiles pertaining to the test tissue sample, wherein the sampled subset of image tiles is sampled from the image tiles pertaining to the test tissue sample.

### Claim 12 (depends on 10)

The storage medium of claim 10,
wherein generating the tile feature vector for each image tile comprises applying a tile featurization model to each image tile,
wherein generating the aggregate feature vector comprises applying an attention model to the tile feature vectors of the sampled subset of image tiles, wherein the attention model is configured to determine, for each tile feature vector of the sampled subset, an attention weight based on each tile feature vector and to sum products of each attention weight and the respective tile feature vector to produce the aggregate feature vector, and
wherein the featurization model, the attention model, and the machine-learned prediction model are trained synchronously using the first set of H&E stain images from the first set of tissue samples having the positive hormone receptor status and the second set of H&E stain images from the second set of tissue samples having the negative hormone receptor status.

### Claim 13 (depends on 10)

The storage medium of claim 10, the operations further comprising:
sampling a second subset of image tiles from the plurality of image tiles of the test H&E stain image;
generating a tile feature vector for each image tile of the second sampled subset of image tiles;
generating a second aggregate feature vector for the test H&E stain image by aggregating the tile feature vectors of the image tiles of the second sampled subset; and
predicting a second hormone receptor status by applying the machine-learned prediction model to the second aggregate feature vector, wherein a cumulative prediction of the hormone receptor status combines the hormone receptor status and the second hormone receptor status.

### Claim 14

A method for training one or more machine-learned models configured to predict hormone receptor status comprising:
accessing a first set of H&E stain images from a first set of tissue samples having a positive hormone receptor status and a second set of H&E stain images from a second set of tissue samples having a negative hormone receptor status;
partitioning each H&E stain image in the first set of H&E stain images and the second set of H&E stain images into a plurality of image tiles;
for each partitioned H&E stain image, sampling a subset of image tiles from the plurality of image tiles;
initializing a tile featurization model configured to generate a feature vector for an image tile, an attention model configured to generate an aggregate feature vector for a H&E stain image by aggregating feature vectors for a sampled subset of image tiles from the H&E stain image, and a prediction model configured to generate a prediction of hormone receptor status based on an aggregate feature vector; and
training the tile featurization model, the attention model, and the prediction model using the first set of H&E stain images and the second set of H&E stain images.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
excluding a portion of one or more H&E stain images determined to be non-relevant to hormone receptor status, wherein the sampled subset of image tiles does not include image tiles from the excluded portions of H&E stain images.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
overwriting an image tile in a sampled subset with a mean pixel intensity of the H&E stain image from which the image tile is partitioned.

### Claim 17 (depends on 11)

The method of claim 11, wherein at least one of the tile featurization model, the attention model, and the prediction model is a neural network trained via backpropagation.

### Claim 18 (depends on 11)

The method of claim 11, wherein training the tile featurization model, the attention model, and the prediction model occurs in batches of sampled subsets, wherein each batch of sampled subsets is trained over a plurality of epochs.

### Claim 19 (depends on 11)

The method of claim 11, wherein training the prediction model further comprises:
in a first epoch, identifying one or more sampled subsets predicted incorrectly by the prediction model; and
including the one or more sampled subsets predicted incorrectly in a second epoch subsequent to the first epoch.

### Claim 20 (depends on 11)

The method of claim 11, further comprising applying, to a sampled subset, one of:
a rotation regularization which rotates one or more image tiles;
a mirror regularization which mirror flips one or more image tiles; and
a color jittering regularization which jitters one or more colors of one or more image tiles.
