# US 2024/0257910 A1 · Method and electronic device for predicting patch-level gene expression from histology image by using artificial intelligence model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257910 A1
- Document ID: US20240257910A1
- Status: Published application
- Kind: A1
- Application number: 18/425,291
- Filed: 2024-01-29
- Published: 2024-08-01
- Earliest claimed: 2023-01-30
- Assignees: Research & Business Foundation Sungkyunkwan University
- Applicants: Research & Business Foundation Sungkyunkwan University
- Inventors: Young Min CHUNG; Kyeong Chan IM; Joo Sang LEE
- Web page: https://patentgrep.com/patent/US20240257910A1

## CPC

- G16B 25/10: Gene or protein expression profiling; Expression-ratio estimation or normalisation
- G16B 40/10: Signal processing, e.g. from mass spectrometry [MS] or from PCR
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16B: BIOINFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR GENETIC OR PROTEIN-RELATED DATA PROCESSING IN COMPUTATIONAL MOLECULAR BIOLOGY
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A method, performed by an electronic device, of predicting gene expression from a histology image by using an artificial intelligence model may include identifying a first patch of the histology image divided into a plurality of patches, initial feature data of the first patch, and initial feature data of a second patch of the histology image, extracting global feature data of the first patch based on the initial feature data of the first patch and the initial feature data of the second patch, by using a first artificial intelligence model, extracting local feature data of the first patch from the first patch by using a second artificial intelligence model, and predicting a gene expression value for the first patch based on the global feature data of the first patch and the local feature data of the first patch, by using a third artificial intelligence model.

## Claims

### Claim 1

A method, performed by an electronic device, of predicting gene expression from a histology image by using an artificial intelligence model, the method comprising:
identifying a first patch of the histology image divided into a plurality of patches, initial feature data of the first patch, and initial feature data of a second patch of the histology image;
extracting global feature data of the first patch based on the initial feature data of the first patch and the initial feature data of the second patch, by using a first artificial intelligence model;
extracting local feature data of the first patch from the first patch by using a second artificial intelligence model; and
predicting a gene expression value for the first patch based on the global feature data of the first patch and the local feature data of the first patch, by using a third artificial intelligence model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the identifying of the first patch of the histology image divided into the plurality of patches, the initial feature data of the first patch, and the initial feature data of the second patch of the histology image comprises extracting the initial feature data of the first patch from the first patch, and extracting the initial feature data of the second patch from the second patch, by using a pre-trained artificial intelligence model.

### Claim 3 (depends on 1)

The method of claim 1, wherein the predicting of the gene expression value for the first patch comprises:
generating local-global feature data of the first patch by concatenating the global feature data of the first patch with the local feature data of the first patch; and
predicting the gene expression value for the first patch based on the local-global feature data of the first patch, by using the third artificial intelligence model.

### Claim 4 (depends on 1)

The method of claim 1, wherein the extracting of the local feature data of the first patch from the first patch by using the second artificial intelligence model comprises extracting first local feature data and second local feature data of the first patch from the first patch, by using the second artificial intelligence model, and
the predicting of the gene expression value for the first patch comprises:
generating first local-global feature data in which the first local feature data of the first patch is concatenated with the global feature data of the first patch, and second local-global feature data in which the second local feature data of the first patch is concatenated with the global feature data of the first patch;
predicting a first gene expression value for the first patch based on the first local-global feature data, and predicting a second gene expression value for the first patch based on the second local-global feature data, by using the third artificial intelligence model; and
determining a final gene expression value for the first patch based on at least one of the first gene expression value or the second gene expression value.

### Claim 5 (depends on 4)

The method of claim 4, wherein the second artificial intelligence model comprises a plurality of sequentially connected layers,
the first local feature data is output data from a deepest layer among the plurality of layers, and
the second local feature data is data generated based on output data from a layer other than the deepest layer among the plurality of layers.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first artificial intelligence model comprises a positional information encoder configured to encode patch positional information in the histology image, and
the extracting of the global feature data of the first patch based on the initial feature data of the first patch and the initial feature data of the second patch, by using the first artificial intelligence model comprises extracting the global feature data of the first patch in which positional information of the first patch is encoded, by using the positional information encoder.

### Claim 7 (depends on 6)

The method of claim 6, wherein the extracting of the global feature data of the first patch in which the positional information of the first patch is encoded, by using the positional information encoder comprises:
performing at least one self-attention operation based on the initial feature data of the first patch and the initial feature data of the second patch;
performing a convolution operation on result data of the at least one self-attention operation, based on the positional information of the first patch and positional information of the second patch;
performing a deformable convolution operation on the result data of the at least one self-attention operation, based on the positional information of the first patch and the positional information of the second patch; and
extracting the global feature data of the first patch in which the positional information of the first patch is encoded, based on result data of the convolution operation and result data of the deformable convolution operation.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model are connected to each other in an end-to-end manner, and trained simultaneously.

### Claim 9 (depends on 1)

The method of claim 1, wherein the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model are trained by using a loss function based on a difference between a gene expression value predicted for a target patch included in a training image, and a ground-truth gene expression value for the target patch.

### Claim 10 (depends on 1)

The method of claim 1, wherein the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model are trained by using a loss function based on a difference between first local-global feature data for a target patch included in a training image, and second local-global feature data for the target patch,
the first local-global feature data for the target patch is generated by concatenating first local feature data of the target patch, which is output data from a deepest layer of the second artificial intelligence model for the target patch, with global feature data of the target patch, and
the second local-global feature data for the target patch is generated by concatenating second local feature data of the target patch based on output data from a layer other than the deepest layer of the second artificial intelligence model for the target patch, with the global feature data of the target patch.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model are trained by using a loss function based on a difference between a first gene expression value for a target patch included in a training image, and a second gene expression value for the target patch,
the first gene expression value for the target patch is predicted based on output data from a deepest layer of the second artificial intelligence model for the target patch, and
the second gene expression value for the target patch is predicted based on output data from a layer other than the deepest layer of the second artificial intelligence model for the target patch.

### Claim 12 (depends on 1)

A non-transitory computer-readable recording medium having recorded thereon a program for executing, on a computer, the method of claim 1.

### Claim 13

An electronic device for predicting gene expression from a histology image by using an artificial intelligence model, the electronic device comprising:
a memory storing one or more instructions; and
at least one processor configured to identify a first patch of the histology image divided into a plurality of patches, initial feature data of the first patch, and initial feature data of a second patch of the histology image, extract global feature data of the first patch based on the initial feature data of the first patch and the initial feature data of the second patch, by using a first artificial intelligence model, extract local feature data of the first patch from the first patch by using a second artificial intelligence model, and predict a gene expression value for the first patch based on the global feature data of the first patch and the local feature data of the first patch, by using a third artificial intelligence model.

### Claim 14 (depends on 13)

The electronic device of claim 13, wherein the at least one processor is further configured to extract the initial feature data of the first patch from the first patch, and extract the initial feature data of the second patch from the second patch, by using a pre-trained artificial intelligence model.

### Claim 15 (depends on 13)

The electronic device of claim 13, wherein the at least one processor is further configured to generate local-global feature data of the first patch by concatenating the global feature data of the first patch with the local feature data of the first patch, and predict the gene expression value for the first patch based on the local-global feature data of the first patch, by using the third artificial intelligence model.

### Claim 16 (depends on 13)

The electronic device of claim 13, wherein the at least one processor is further configured to extract first local feature data and second local feature data of the first patch from the first patch, by using the second artificial intelligence model, generate first local-global feature data in which the first local feature data of the first patch is concatenated with the global feature data of the first patch, and second local-global feature data in which the second local feature data of the first patch is concatenated with the global feature data of the first patch, predict a first gene expression value for the first patch based on the first local-global feature data, and predict a second gene expression value for the first patch based on the second local-global feature data, by using the third artificial intelligence model, and determine a final gene expression value for the first patch based on at least one of the first gene expression value or the second gene expression value.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein the second artificial intelligence model comprises a plurality of sequentially connected layers,
the first local feature data is output data from a deepest layer among the plurality of layers, and
the second local feature data is data generated based on output data from a layer other than the deepest layer among the plurality of layers.

### Claim 18 (depends on 13)

The electronic device of claim 13, wherein the first artificial intelligence model comprises a positional information encoder configured to encode patch positional information in the histology image, and
the at least one processor is further configured to extract the global feature data of the first patch in which positional information of the first patch is encoded, by using the positional information encoder.

### Claim 19 (depends on 18)

The electronic device of claim 18, wherein the at least one processor is further configured to perform at least one self-attention operation based on the initial feature data of the first patch and the initial feature data of the second patch, perform a convolution operation on result data of the at least one self-attention operation, based on the positional information of the first patch and positional information of the second patch, perform a deformable convolution operation on the result data of the at least one self-attention operation, based on the positional information of the first patch and the positional information of the second patch, and extract the global feature data of the first patch in which the positional information of the first patch is encoded, based on result data of the convolution operation and result data of the deformable convolution operation.

### Claim 20 (depends on 13)

The electronic device of claim 13, wherein
the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model are trained by using a loss function based on a difference between a first gene expression value for a target patch included in a training image, and a second gene expression value for the target patch,
the first gene expression value for the target patch is predicted based on output data from a deepest layer of the second artificial intelligence model for the target patch, and
the second gene expression value for the target patch is predicted based on output data from a layer other than the deepest layer of the second artificial intelligence model for the target patch.
