# US 12,223,642 B2 · Deep learning model to predict data from an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,642 B2
- Document ID: US12223642B2
- Status: Granted
- Kind: B2
- Application number: 17/306,170
- Filed: 2021-05-03
- Published: 2022-11-03
- Granted: 2025-02-11
- Earliest claimed: 2021-05-03
- Assignees: Rakuten Group, Inc.
- Applicants: Rakuten Group, Inc.
- Inventors: Shreya Sharma; Abhishek Vahadane; Srikanth Ragothaman; Shantanu Majumdar
- Web page: https://patentgrep.com/patent/US12223642B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 7/73: using feature-based methods
- G06V 20/698: {Matching; Classification}
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06V 10/82: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, computer program, and computer system is provided for predicting data from an image. An image is divided into one or more patch images. Spatial features corresponding to the one or more patch images are compressed. Output data corresponding to the compressed spatial features is predicted. The output data is predicted based on minimizing one or more loss functions corresponding to the compressed spatial features.

## Claims

### Claim 1

A method of predicting data from an image, executable by a processor, comprising:
dividing a hematoxylin and eosin (H&E) image into one or more patch images;
compressing, using a first model, spatial features corresponding to the one or more patch images; and
predicting, using a second model, output molecular data corresponding to the compressed spatial features; and
adjusting parameters of the first model based on minimizing a first loss function corresponding to the compressed spatial features and adjusting parameters of the second model based on minimizing a second loss function corresponding to the compressed spatial features,
wherein the first loss function comprises a regression loss function corresponding to compressed feature expression images output by the first model based on the one or more patch images, and
wherein the second loss function comprises a feature compression loss function corresponding to second output data, output by the second model, indicated by the H&E image based on the compressed feature expression images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the compressing the spatial features comprises:
determining spatial arrangements corresponding to the compressed spatial features; and
generating a deep learning model based on the determined spatial arrangements.

### Claim 3 (depends on 1)

The method of claim 1, wherein the compressing the spatial features comprises:
detecting a pathology associated with the H&E image;
identifying patch images corresponding to the detected pathology;
identifying normal patch images associated with the H&E image;
determining spatial arrangements corresponding to the normal patch images and the patch images corresponding to the detected pathology; and
generating or training a deep learning model based on the determined spatial arrangements.

### Claim 4 (depends on 1)

The method of claim 1, wherein a patch image from among the one or more patch images comprises at least a portion of the H&E image and data corresponding to a location of the spatial features within the H&E image.

### Claim 5

A computer system for predicting data from an image, the computer system comprising:
one or more computer-readable non-transitory storage media configured to store computer program code; and
one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
dividing code configured to cause the one or more computer processors to divide a hematoxylin and eosin (H&E) image into one or more patch images;
compressing code of a first model configured to cause the one or more computer processors to compress spatial features corresponding to the one or more patch images;
predicting code of a second model configured to cause the one or more computer processors to predict output molecular data corresponding to the compressed spatial features; and
adjust parameters of the first model based on minimizing a first loss function corresponding to the compressed spatial features and adjust parameters of the second model based on minimizing a second loss function corresponding to the compressed spatial features,
wherein the first loss function comprises a regression loss function corresponding to compressed feature expression images output by the first model based on the one or more patch images, and
wherein the second loss function comprises a feature compression loss function corresponding to second output data, output by the second model, indicated by the H&E image based on the compressed feature expression images.

### Claim 6 (depends on 5)

The computer system of claim 5, wherein the compressing code comprises:
determining code configured to cause the one or more computer processors to determine spatial arrangements corresponding to the compressed spatial features; and
generating code configured to cause the one or more computer processors to generate a deep learning model based on the determined spatial arrangements.

### Claim 7 (depends on 5)

The computer system of claim 5, wherein the compressing code comprises:
detecting code configured to cause the one or more computer processors to detect a pathology associated with the H&E image;
first identifying code configured to cause the one or more computer processors to identify patch images corresponding to the detected pathology;
second identifying code configured to cause the one or more computer processors to identify normal patch images associated with the H&E image;
determining code configured to cause the one or more computer processors to determine spatial arrangements corresponding to the normal patch images and the patch images corresponding to the detected pathology; and
generating code configured to cause the one or more computer processors to generate a deep learning model based on the determined spatial arrangements.

### Claim 8 (depends on 5)

The computer system of claim 5, wherein a patch image from among the one or more patch images comprises at least a portion of the H&E image and data corresponding to a location of the spatial features within the H&E image.

### Claim 9

A non-transitory computer readable medium having stored thereon a computer program for predicting data from an image, the computer program configured to cause one or more computer processors to:
divide a hematoxylin and eosin (H&E) image into one or more patch images;
compress, using a first model, spatial features corresponding to the one or more patch images;
predict, using a second model, output molecular data corresponding to the compressed spatial features; and
adjust parameters of the first model based on minimizing a first loss function corresponding to the compressed spatial features and adjusting parameters of the second model based on minimizing a second loss function corresponding to the compressed spatial features,
wherein the first loss function comprises a regression loss function corresponding to compressed feature expression images output by the first model based on the one or more patch images, and
wherein the second loss function comprises a feature compression loss function corresponding to second output data, output by the second model, indicated by the H&E image based on the compressed feature expression images.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein the computer program is further configured to cause one or more computer processors to: determine spatial arrangements corresponding to the compressed spatial features; and generate a deep learning model based on the determined spatial arrangements.

### Claim 11 (depends on 9)

The non-transitory computer readable medium of claim 9, wherein the computer program is further configured to cause one or more computer processors to: detect a pathology associated with the H&E image; identify patch images corresponding to the detected pathology; identify normal patch images associated with the H&E image; determine spatial arrangements corresponding to the normal patch images and the patch images corresponding to the detected pathology; and generate a deep learning model based on the determined spatial arrangements.