# US 2023/0410477 A1 · Method and device for segmenting objects in images using artificial intelligence

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410477 A1
- Document ID: US20230410477A1
- Status: Published application
- Kind: A1
- Application number: 18/085,046
- Filed: 2022-12-20
- Published: 2023-12-21
- Earliest claimed: 2022-06-15
- Assignees: University-Industry Foundation, Yonsei University
- Applicants: University Industry Foundation, Yonsei University
- Inventors: Hyeran Byun; Sanghuk Lee; Cheolhyun Mun; Jewook Lee; Pilhyeon Lee
- Web page: https://patentgrep.com/patent/US20230410477A1

## CPC

- G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes}
- G06V 10/765: {using rules for classification or partitioning the feature space}
- G06T 7/11: Region-based segmentation
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method of segmenting objects in an image using artificial intelligence includes obtaining, by an analysis device, an image containing at least one object; inputting the image acquired by the analysis device into a segmentation model; and segmenting, by the analysis device, objects in the acquired image based on values output by the segmentation model, wherein an image containing at least one object is used as learning data, the size of objects in the learning data is estimated as part of the learning process, different weightings are given to pixels of which the objects consist according to the estimated size of the objects, and the segmentation model is trained based on a loss function in which the given weightings are considered, a size-weighted loss function.

## Claims

### Claim 1

A method of segmenting objects in an image using artificial intelligence, comprising:
obtaining, by an analysis device, an image containing at least one object;
inputting the image acquired by the analysis device into a segmentation model; and
segmenting, by the analysis device, objects in the acquired image based on values output by the segmentation model,
wherein an image containing at least one object is used as learning data, the size of objects in the learning data is estimated as part of the learning process, different weightings are given to pixels of which the objects consist according to the estimated size of the objects, and the segmentation model is trained based on a loss function in which the given weightings are considered, a size-weighted loss function.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the method of giving different weightings to pixels involves giving high weightings to pixels when the estimated relative size of the object is large and giving low weightings to pixels when the estimated relative size of the object is small.

### Claim 3 (depends on 1)

The method of claim 1,
wherein an upper limit for weightings is set, and a weighting is given within the range not exceeding the upper limit.

### Claim 4 (depends on 1)

The method of claim 1,
wherein the loss function in which weightings are considered is based on L S according to the following equations:
L
sw
=
-
1
H
×
W
⁢
∑
c
=
1
C
∑
x
=
1
H
∑
y
=
1
W
Y
c
,
x
,
y
⁢
w
x
,
y
⁢
log
⁡
(
p
c
,
x
,
y
)
,
[
Equations
]
wherein H is the size of an image, W is the width of the image, Yc,x,y is a class label at corresponding coordinates, Wx,y is a weighting at the corresponding coordinates calculated according to the equation below, and Pc,x,y is the probability that a pixel is expected to belong to the c-th class at the corresponding coordinates, and
w
x
,
y
=
{
1
,
if
⁢
Y
x
,
y
=
0
,
min
⁡
(
τ
,
∑
k
=
1
K
⁢
S
c
,
k
S
c
,
k
)
,
otherwise
,
,
wherein x and y are x and y coordinates of a pixel, respectively, Yx,y is a class label at the x and y coordinates, Sc,k is the number of pixels in its connected component, i.e., the k-th instance in the c-th class, k is the number of an instance in a corresponding class, and r is an upper limit for weightings.

### Claim 5 (depends on 1)

The method of claim 1,
wherein the method of training a learning model based on the loss function in which weightings are considered involves the step of first training the learning model using an image including an object having a size smaller than a predetermined reference value in the learning data; and the step of making a final learning model by continuously training the trained model using an image including an object having a size greater than the predetermined reference value in the learning data.

### Claim 6 (depends on 1)

The method of claim 1 further comprising the step of evaluating the performance of the segmentation model by the analysis device, which follows the step of segmenting objects in an image.

### Claim 7 (depends on 6)

The method of claim 6,
wherein the step of evaluating the performance of the segmentation model involves the step of detecting a class and objects belonging to the class in an image including at least one object; and the step of calculating a value of intersection over union (IoU) for each of the objects and evaluating the performance of the segmentation model based on the IoU value per object.

### Claim 8 (depends on 6)

The method of claim 6,
wherein the step of evaluating the performance of the segmentation model involves using a value of instance aware mean intersection over union (IA-mIoU) according to the following equation:
IoU
c
,
i
=
Pr
c
,
i
⋂
GT
c
,
i
Pr
c
,
i
⋃
GT
c
,
i
,
IoU
^
c
=
∑
i
=
0
T
⁢
IoU
c
,
i
T
[
Equation
]
IA
-
mIoU
=
∑
c
=
0
N
⁢
IoU
^
c
N
,
wherein c is the number of a class, i is the number of an object in the class, T is the total number of objects in the class, Prc,i is a value of a prediction mask of the i-th object in the c-th class, GTc,i is a value of a ground-truth mask of the i-th object in the c-th class, IoUc,i is an IoU value of the i-th object in the c-th class, Io{circumflex over ( )}Uc is the average value of the IoUc,i in the c-th class, and IA-mIoU is the average of the Io{circumflex over ( )}Uc of all classes in an image.

### Claim 9

A device for segmenting objects in an image using artificial intelligence, comprising:
an input device for obtaining an image including at least one object;
a storage device for storing a segmentation model that segments objects in the image acquired by the input device; and
an arithmetic device inputting the image acquired by the input device to the segmentation model and segmenting objects in the image based on values output by the segmentation model,
wherein an image containing at least one object is used as learning data, the size of objects in the learning data is estimated as part of the learning process, different weightings are given to pixels of which the objects consist according to the estimated size of the objects, and the segmentation model is trained based on a loss function in which the given weightings are considered, a size-weighted loss function.