# US 2024/0005507 A1 · Image processing method, apparatus and non-transitory computer readable medium for performing image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005507 A1
- Document ID: US20240005507A1
- Status: Published application
- Kind: A1
- Application number: 18/046,385
- Filed: 2022-10-13
- Published: 2024-01-04
- Earliest claimed: 2022-05-24
- Assignees: Alibaba (China) Co., Ltd.
- Applicants: Alibaba (China) Co., Ltd.
- Inventors: Jiawen YAO; Yingda XIA; Ke YAN; Dakai JIN; Xiansheng HUA; Le LU; Ling ZHANG
- Web page: https://patentgrep.com/patent/US20240005507A1

## CPC

- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06V 10/82: using neural networks
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/73: using feature-based methods
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V 2201/031: of internal organs
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing method is provided. The method includes obtaining a to-be-processed image comprising a target object, and inputting the to-be-processed image to a convolutional layer of an image processing model, to obtain an initial feature map of the to-be-processed image, wherein the image processing model comprises an encoder and a decoder; inputting the initial feature map to a self-attention mechanism layer of the encoder, and obtaining a target feature map corresponding to the initial feature map according to position information of each feature in the initial feature map and a position relationship between the each feature and other features; and inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image

## Claims

### Claim 1

An image processing method, comprising:
obtaining a to-be-processed image comprising a target object;
inputting the to-be-processed image to a convolutional layer of an image processing model to obtain an initial feature map of the to-be-processed image, wherein the image processing model comprises an encoder and a decoder;
inputting the initial feature map to a self-attention mechanism layer of the encoder to obtain a target feature map corresponding to the initial feature map according to position information of each feature in the initial feature map and a position relationship between the each feature and other features; and
inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image.

### Claim 2 (depends on 1)

The image processing method according to claim 1, wherein the image processing model is trained by:
obtaining a sample image comprising a sample object, and determining a sample object segmentation map and a sample object label of the sample object in the sample image;
inputting the sample image to a convolutional layer of the encoder, to obtain an initial sample feature map of the sample image;
inputting the initial sample feature map to the self-attention mechanism layer of the encoder;
obtaining a target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and a position relationship between the each initial sample feature and other initial sample features;
inputting the target sample feature map to the decoder for processing, to obtain a predicted object segmentation map and a predicted object label of the sample object in the sample image; and
training the image processing model according to a loss function composed of the sample object segmentation map and the sample object label, and the predicted object segmentation map and the predicted object label.

### Claim 3 (depends on 2)

The image processing method according to claim 2, wherein obtaining the target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and a position relationship between the each initial sample feature and other initial sample features comprises:
determining a coordinate grid of the initial sample feature map, and determining position information of each initial sample feature in the initial sample feature map according to the coordinate grid;
calculating, by using a first preset algorithm, a target sample feature corresponding to the each initial sample feature according to the position information of the each initial sample feature and the position relationship between the each initial sample feature and the other initial sample features; and
obtaining the target sample feature map corresponding to the initial sample feature map according to the target sample feature.

### Claim 4 (depends on 2)

The image processing method according to claim 2, wherein obtaining the target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and a position relationship between the each initial sample feature and other initial sample features comprises:
determining a coordinate grid of the initial sample feature map, and determining position information of each initial sample feature in the initial sample feature map according to the coordinate grid;
determining an associated region of the each initial sample feature and an associated sample feature of the associated region according to the position information;
calculating, by using a second preset algorithm, a target sample feature corresponding to the each initial sample feature according to the position information of the each initial sample feature and a position relationship between the each initial sample feature and the associated sample feature; and
obtaining the target sample feature map corresponding to the initial sample feature map according to the target sample feature.

### Claim 5 (depends on 4)

The image processing method according to claim 4, wherein determining the associated region of the each initial sample feature and the associated sample feature of the associated region according to the position information comprises:
determining an associated distance between the each initial sample feature and the other initial sample features according to the position information of the each initial sample feature; and
determining the associated region of the each initial sample feature according to the associated distance, and determining each initial sample feature in the associated region as the associated sample feature.

### Claim 6 (depends on 4)

The image processing method according to claim 4, wherein the calculating, by using the second preset algorithm, the target sample feature corresponding to the each initial sample feature according to the position information of the each initial sample feature and the position relationship between the each initial sample feature and the associated sample feature comprises:
calculating, by using the second preset algorithm, the target sample feature corresponding to the each initial sample feature according to the position information of the each initial sample feature and an associated distance between the each initial sample feature and the associated sample feature.

### Claim 7 (depends on 2)

The image processing method according to claim 2, wherein before obtaining the target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and the position relationship between the each initial sample feature and other initial sample features, the method further comprises:
performing dimension reduction on the initial sample feature map according to a preset dimension reduction method, to obtain a dimensionally reduced initial sample feature map.

### Claim 8 (depends on 2)

The image processing method according to claim 2, wherein the image processing model least comprises two self-attention mechanism layers, and each self-attention mechanism layer is arranged behind the convolutional layer.

### Claim 9 (depends on 2)

The image processing method according to claim 2, wherein after obtaining a to-be-processed image of a target object, the method further comprises:
inputting the to-be-processed image to the image processing model, to obtain the object segmentation map and the object label of the target object in the to-be-processed image.

### Claim 10 (depends on 1)

The image processing method according to claim 1, wherein obtaining the object segmentation map and the object label of the target object in the to-be-processed image comprises:
obtaining the object segmentation map of the target object in the to-be-processed image, and a segmentation label of each segmentation block in the object segmentation map; and
calculating volumes of segmentation blocks having the same segmentation label, and
determining the object label of the target object according to volumes of segmentation blocks corresponding to each type of segmentation labels.

### Claim 11 (depends on 2)

The image processing method according to claim 2, wherein determining the sample object segmentation map and the sample object label of the sample object in the sample image comprises:
determining at least two contrast enhanced sample images comprising the sample object; and
determining the sample object segmentation map and the sample object label of the sample object in the sample image according to a sample annotation of the sample object in the at least two contrast enhanced sample images.

### Claim 12 (depends on 1)

The image processing method according to claim 1, wherein the target object is an esophagus, the to-be-processed image is a computed tomography (CT) image, and the object label is a pathological label.

### Claim 13 (depends on 1)

The image processing method according to claim 1, wherein obtaining the to-be-processed image comprising a target object comprises:
receiving a processing request for an esophageal CT image from a user, wherein the esophageal CT image is an image of an esophagus scanned by an electronic computed tomography scanner and the esophageal CT image is the to-be-processed image; and
after inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image further comprises:
outputting and displaying, in a case that the user is determined as a first type of user, to the first type of user a segmentation map and a pathological label of the esophagus in the esophageal CT image, or
outputting and displaying, in a case that the user is determined as a second type of user, to the second type of user the pathological label of the esophageal CT image.

### Claim 14

An apparatus for performing image processing, the apparatus comprising:
a memory configured to store instructions; and
one or more processors configured to execute the instructions to cause the apparatus to perform:
obtaining a to-be-processed image comprising a target object;
inputting the to-be-processed image to a convolutional layer of an image processing model to obtain an initial feature map of the to-be-processed image, wherein the image processing model comprises an encoder and a decoder;
inputting the initial feature map to a self-attention mechanism layer of the encoder to obtain a target feature map corresponding to the initial feature map according to position information of each feature in the initial feature map and a position relationship between the each feature and other features; and
inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image.

### Claim 15 (depends on 14)

The apparatus according to claim 14, wherein the one or more processors are further configured to execute the instructions to cause the apparatus to perform:
obtaining a sample image comprising a sample object, and determining a sample object segmentation map and a sample object label of the sample object in the sample image;
inputting the sample image to a convolutional layer of the encoder, to obtain an initial sample feature map of the sample image;
inputting the initial sample feature map to the self-attention mechanism layer of the encoder;
obtaining a target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and a position relationship between the each initial sample feature and other initial sample features;
inputting the target sample feature map to the decoder for processing, to obtain a predicted object segmentation map and a predicted object label of the sample object in the sample image; and
training the image processing model according to a loss function composed of the sample object segmentation map and the sample object label, and the predicted object segmentation map and the predicted object label.

### Claim 16 (depends on 14)

The apparatus according to claim 14, wherein the target object is an esophagus, the to-be-processed image is a computed tomography (CT) image, and the object label is a pathological label.

### Claim 17 (depends on 14)

The apparatus according to claim 14, wherein the one or more processors are further configured to execute the instructions to cause the apparatus to perform:
receiving a processing request for an esophageal CT image from a user, wherein the esophageal CT image is an image of an esophagus scanned by an electronic computed tomography scanner and the esophageal CT image is the to-be-processed image; and
after inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image further comprises:
outputting and displaying, in a case that the user is determined as a first type of user, to the first type of user a segmentation map and a pathological label of the esophagus in the esophageal CT image, or
outputting and displaying, in a case that the user is determined as a second type of user, to the second type of user the pathological label of the esophageal CT image.

### Claim 18

A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to perform:
obtaining a to-be-processed image comprising a target object;
inputting the to-be-processed image to a convolutional layer of an image processing model to obtain an initial feature map of the to-be-processed image, wherein the image processing model comprises an encoder and a decoder;
inputting the initial feature map to a self-attention mechanism layer of the encoder to obtain a target feature map corresponding to the initial feature map according to position information of each feature in the initial feature map and a position relationship between the each feature and other features; and
inputting the target feature map to the decoder for processing, to obtain an object segmentation map and an object label of the target object in the to-be-processed image.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to further perform:
obtaining a sample image comprising a sample object, and determining a sample object segmentation map and a sample object label of the sample object in the sample image;
inputting the sample image to a convolutional layer of the encoder, to obtain an initial sample feature map of the sample image;
inputting the initial sample feature map to the self-attention mechanism layer of the encoder;
obtaining a target sample feature map corresponding to the initial sample feature map according to position information of each initial sample feature in the initial sample feature map and a position relationship between the each initial sample feature and other initial sample features;
inputting the target sample feature map to the decoder for processing, to obtain a predicted object segmentation map and a predicted object label of the sample object in the sample image; and
training the image processing model according to a loss function composed of the sample object segmentation map and the sample object label, and the predicted object segmentation map and the predicted object label.

### Claim 20 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the target object is an esophagus, the to-be-processed image is a computed tomography (CT) image, and the object label is a pathological label.
