# US 2021/0343016 A1 · Medical image processing method and apparatus, electronic medical device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343016 A1
- Document ID: US20210343016A1
- Status: Published application
- Kind: A1
- Application number: 17/367,280
- Filed: 2021-07-02
- Published: 2021-11-04
- Earliest claimed: 2019-05-22
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Kuan TIAN; Cheng JIANG
- Web page: https://patentgrep.com/patent/US20210343016A1

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06T 7/11: Region-based segmentation
- G06T 7/62: of area, perimeter, diameter or volume
- G06V 10/17: {using hand-held instruments}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/82: using neural networks
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 30/19153: {using rules for classification or partitioning the feature space}
- G06V 30/19173: {Classification techniques}
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06T 2207/10116: X-ray image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30068: Mammography; Breast
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V 2201/032: of protuberances, polyps nodules, etc.
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A medical image processing method includes: obtaining a biological tissue image including a biological tissue, recognizing, in the biological tissue image, a first region of a lesion object in the biological tissue; recognizing a lesion attribute matching the lesion object; dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions; obtaining target quadrant position information of a quadrant region in which the first region is located; and generating medical service data according to the target quadrant position information and the lesion attribute.

## Claims

### Claim 1

A medical image processing method, performed by an electronic device, the method comprising:
obtaining a biological tissue image comprising a biological tissue;
recognizing, in the biological tissue image, a first region of a lesion object in the biological tissue, and recognizing a lesion attribute matching the lesion object;
dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions;
obtaining target quadrant position information of a quadrant region in which the first region is located; and
generating medical service data according to the target quadrant position information and the lesion attribute.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the biological tissue comprises a first tissue object; and
the dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions comprises:
recognizing a second region of the first tissue object in the biological tissue image;
determining quadrant segmentation lines in the biological tissue image according to the second region; and
using the image region of the biological tissue in the biological tissue image as a tissue image region, and dividing the tissue image region into the plurality of quadrant regions according to the quadrant segmentation lines.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the biological tissue image comprises a first biological tissue image and a second biological tissue image, and the first biological tissue image and the second biological tissue image are images of the biological tissue in different views; and
the recognizing a second region of the first tissue object in the biological tissue image comprises:
obtaining an image semantic segmentation model, and determining a first marked region of the first tissue object in the first biological tissue image based on the image semantic segmentation model;
determining a second marked region of the first tissue object in the second biological tissue image based on the image semantic segmentation model; and
determining the first marked region and the second marked region as the second region.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the determining a first marked region of the first tissue object in the first biological tissue image based on the image semantic segmentation model comprises:
performing forward convolution and backward convolution on the first biological tissue image based on a forward convolutional layer and a transposed convolutional layer in the image semantic segmentation model, to obtain a convolutional feature map;
determining an object attribute of each pixel in the first biological tissue image according to the convolutional feature map, the object attribute comprising a first tissue attribute indicating whether a corresponding pixel pertains to the first tissue object; and
using an image region formed by pixels pertaining to the first tissue object as the first marked region of the first tissue object in the first biological tissue image.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the quadrant segmentation lines comprise a first segmentation line corresponding to the first biological tissue image and a second segmentation line corresponding to the second biological tissue image, and the object attribute further comprises a second tissue attribute; and
the determining quadrant segmentation lines in the biological tissue image according to the second region comprises:
obtaining an edge boundary of the biological tissue in the first biological tissue image, and determining the first segmentation line in the first biological tissue image according to the first marked region and the edge boundary;
using an image region formed by pixels pertaining to the second tissue attribute as an object region, of a second tissue object in the second biological tissue image, in the second biological tissue image; and
determining an object boundary of the object region, and determining the second segmentation line in the second biological tissue image according to the second marked region and the object boundary.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the first biological tissue image is an image of the biological tissue in a craniocaudal view, and the second biological tissue image is an image of the biological tissue in a mediolateral oblique view; and
the using the image region of the biological tissue in the biological tissue image as a tissue image region, and dividing the tissue image region into the plurality of quadrant regions according to the quadrant segmentation lines comprises:
using an image region of the biological tissue in the first biological tissue image as a first tissue image region, and dividing the first tissue image region into an inner quadrant region and an outer quadrant region in the first biological tissue image according to the first segmentation line;
using an image region of the biological tissue in the second biological tissue image as a second tissue image region, and dividing the second tissue image region into an upper quadrant region and a lower quadrant region in the second biological tissue image according to the second segmentation line; and
determining the inner quadrant region, the outer quadrant region, the upper quadrant region, and the lower quadrant region as the quadrant regions.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the first region comprises a first lesion region of the lesion object in a first biological tissue image and a second lesion region of the lesion object in a second biological tissue image, and the biological tissue image comprises the first biological tissue image and the second biological tissue image; and
the generating medical service data according to the target quadrant position information and the lesion attribute comprises:
extracting a first subimage corresponding to the first lesion region from the first biological tissue image;
extracting a second subimage corresponding to the second lesion region from the second biological tissue image;
obtaining a target matching model, and recognizing a model matching probability between the first subimage and the second subimage based on the target matching model; and
obtaining a condition matching probability between the first lesion region and the second lesion region, and combining the target quadrant position information and the lesion attribute into the medical service data when the model matching probability and the condition matching probability meet a lesion matching condition.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the obtaining a condition matching probability between the first lesion region and the second lesion region comprises:
determining a first size of the first lesion region in the first biological tissue image, determining a second size of the second lesion region in the second biological tissue image, and generating a size matching probability according to the first size and the second size;
determining a first region distance between the first lesion region and an image region corresponding to a first tissue object in the first biological tissue image, and determining a second region distance between the second lesion region and an image region corresponding to the first tissue object in the second biological tissue image;
generating a distance matching probability according to the first region distance and the second region distance; and
determining the size matching probability and the distance matching probability as the condition matching probability.

### Claim 9 (depends on 7)

The method according to claim 7, wherein there are a plurality of lesion objects in the biological tissue image, the model matching probability comprises unit model matching probabilities between first subimages of the plurality of lesion objects and second subimages of the plurality of lesion objects, and the condition matching probability comprises unit condition matching probabilities between first lesion regions of the plurality of lesion objects and second lesion regions of the plurality of lesion objects; and
the combining the target quadrant position information and the lesion attribute into the medical service data when the model matching probability and the condition matching probability meet a lesion matching condition comprises:
selecting a matching probability pair meeting the lesion matching condition from the plurality of unit model matching probabilities and the plurality of unit condition matching probabilities as a target matching probability pair, the target matching probability pair comprising one unit model matching probability and one unit condition matching probability; and
using a lesion object of the target matching probability pair as a target lesion object when the unit model matching probability and the unit condition matching probability in the target matching probability pair meet the lesion matching condition, and combining target quadrant position information of the target lesion object and a lesion attribute of the target lesion object into the medical service data.

### Claim 10 (depends on 7)

The method according to claim 7, further comprising:
obtaining a positive sample image and a negative sample image, the positive sample image comprising a first positive sample image and a second positive sample image, the first positive sample image and the second positive sample image being corresponding to the same lesion object; and the negative sample image comprising a first negative sample image and a second negative sample image, the first negative sample image and the second negative sample image being corresponding to different lesion objects;
obtaining a sample matching model, recognizing a positive sample prediction probability between the first positive sample image and the second positive sample image based on the sample matching model, obtaining a positive sample probability between the first positive sample image and the second positive sample image, and determining a positive sample error between the positive sample prediction probability and the positive sample probability;
recognizing a negative sample prediction probability between the first negative sample image and the second negative sample image based on the sample matching model, obtaining a negative sample probability between the first negative sample image and the second negative sample image, and determining a negative sample error between the negative sample prediction probability and the negative sample probability;
adjusting the sample matching model according to the positive sample error and the negative sample error; and
determining an adjusted sample matching model as the target matching model when the adjusted sample matching model meets a convergence condition.

### Claim 11 (depends on 10)

The method according to claim 10, wherein the obtaining a sample matching model comprises:
obtaining an original matching model, the original matching model being obtained through training according to a non-biological tissue image; and
extracting target model parameters from model parameters comprised in the original matching model, and generating the sample matching model based on transfer learning and the target model parameters.

### Claim 12

A medical image processing apparatus, comprising: a processor and a memory, the processor being connected to the memory, the memory being configured to store a computer program, and the processor being configured to invoke the computer program to:
obtaining a biological tissue image comprising a biological tissue;
recognizing, in the biological tissue image, a first region of a lesion object in the biological tissue, and recognize a lesion attribute matching the lesion object;
dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions, obtaining target quadrant position information of a quadrant region in which the first region is located; and
generating medical service data according to the target quadrant position information and the lesion attribute.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the biological tissue comprises a first tissue object; and
the dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions comprises:
recognizing a second region of the first tissue object in the biological tissue image;
determining quadrant segmentation lines in the biological tissue image according to the second region; and
using the image region of the biological tissue in the biological tissue image as a tissue image region, and dividing the tissue image region into the plurality of quadrant regions according to the quadrant segmentation lines.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the biological tissue image comprises a first biological tissue image and a second biological tissue image, and the first biological tissue image and the second biological tissue image are images of the biological tissue in different views; and
the recognizing a second region of the first tissue object in the biological tissue image comprises:
obtaining an image semantic segmentation model, and determining a first marked region of the first tissue object in the first biological tissue image based on the image semantic segmentation model;
determining a second marked region of the first tissue object in the second biological tissue image based on the image semantic segmentation model; and
determining the first marked region and the second marked region as the second region.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the determining a first marked region of the first tissue object in the first biological tissue image based on the image semantic segmentation model comprises:
performing forward convolution and backward convolution on the first biological tissue image based on a forward convolutional layer and a transposed convolutional layer in the image semantic segmentation model, to obtain a convolutional feature map;
determining an object attribute of each pixel in the first biological tissue image according to the convolutional feature map, the object attribute comprising a first tissue attribute indicating whether a corresponding pixel pertains to the first tissue object; and
using an image region formed by pixels pertaining to the first tissue object as the first marked region of the first tissue object in the first biological tissue image.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the quadrant segmentation lines comprise a first segmentation line corresponding to the first biological tissue image and a second segmentation line corresponding to the second biological tissue image, and the object attribute further comprises a second tissue attribute; and
the determining quadrant segmentation lines in the biological tissue image according to the second region comprises:
obtaining an edge boundary of the biological tissue in the first biological tissue image, and determining the first segmentation line in the first biological tissue image according to the first marked region and the edge boundary;
using an image region formed by pixels pertaining to the second tissue attribute as an object region, of a second tissue object in the second biological tissue image, in the second biological tissue image; and
determining an object boundary of the object region, and determining the second segmentation line in the second biological tissue image according to the second marked region and the object boundary.

### Claim 17 (depends on 16)

The apparatus according to claim 16, wherein the first biological tissue image is an image of the biological tissue in a craniocaudal view, and the second biological tissue image is an image of the biological tissue in a mediolateral oblique view; and
the using the image region of the biological tissue in the biological tissue image as a tissue image region, and dividing the tissue image region into the plurality of quadrant regions according to the quadrant segmentation lines comprises:
using an image region of the biological tissue in the first biological tissue image as a first tissue image region, and dividing the first tissue image region into an inner quadrant region and an outer quadrant region in the first biological tissue image according to the first segmentation line;
using an image region of the biological tissue in the second biological tissue image as a second tissue image region, and dividing the second tissue image region into an upper quadrant region and a lower quadrant region in the second biological tissue image according to the second segmentation line; and
determining the inner quadrant region, the outer quadrant region, the upper quadrant region, and the lower quadrant region as the quadrant regions.

### Claim 18 (depends on 12)

The apparatus according to claim 12, wherein the first region comprises a first lesion region of the lesion object in a first biological tissue image and a second lesion region of the lesion object in a second biological tissue image, and the biological tissue image comprises the first biological tissue image and the second biological tissue image; and
the generating medical service data according to the target quadrant position information and the lesion attribute comprises:
extracting a first subimage corresponding to the first lesion region from the first biological tissue image;
extracting a second subimage corresponding to the second lesion region from the second biological tissue image;
obtaining a target matching model, and recognizing a model matching probability between the first subimage and the second subimage based on the target matching model; and
obtaining a condition matching probability between the first lesion region and the second lesion region, and combining the target quadrant position information and the lesion attribute into the medical service data when the model matching probability and the condition matching probability meet a lesion matching condition.

### Claim 19 (depends on 18)

The apparatus according to claim 18, wherein the obtaining a condition matching probability between the first lesion region and the second lesion region comprises:
determining a first size of the first lesion region in the first biological tissue image, determining a second size of the second lesion region in the second biological tissue image, and generating a size matching probability according to the first size and the second size;
determining a first region distance between the first lesion region and an image region corresponding to a first tissue object in the first biological tissue image, and determining a second region distance between the second lesion region and an image region corresponding to the first tissue object in the second biological tissue image;
generating a distance matching probability according to the first region distance and the second region distance; and
determining the size matching probability and the distance matching probability as the condition matching probability.

### Claim 20

A non-transitory computer storage medium, storing a computer program, the computer program comprising program instructions, the program instructions, when executed by a processor, performing:
obtaining a biological tissue image comprising a biological tissue;
recognizing, in the biological tissue image, a first region of a lesion object in the biological tissue, and recognizing a lesion attribute matching the lesion object;
dividing an image region of the biological tissue in the biological tissue image into a plurality of quadrant regions;
obtaining target quadrant position information of a quadrant region in which the first region is located; and
generating medical service data according to the target quadrant position information and the lesion attribute.
