# US 2026/0087637 A1 · Object segmentation method based on multimodal data fusion and image annotation tool

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087637 A1
- Document ID: US20260087637A1
- Status: Published application
- Kind: A1
- Application number: 19/369,297
- Filed: 2025-10-26
- Published: 2026-03-26
- Earliest claimed: 2024-09-24
- Assignees: Jiangnan University
- Applicants: Jiangnan University
- Inventors: Tianyang XU; Xiaojun WU
- Web page: https://patentgrep.com/patent/US20260087637A1

## CPC

- G06T 7/174: involving the use of two or more images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/11: Region-based segmentation
- G06T 7/337: {involving reference images or patches}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10048: Infrared image
- G06T 2207/20221: Image fusion; Image merging
- G06T 2210/12: Bounding box
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The invention provides an object segmentation method based on multimodal data fusion and an image annotation tool. The method includes acquiring an initial RGB image, an initial infrared image, and an initial depth image that contain an object; aligning the initial RGB image, the initial infrared image, and the initial depth image to obtain a first RGB image, a first infrared image, and a first depth image; specifying an initial prompt point in the first RGB image, and acquiring first masks characterizing an object region from the images in different modalities; fusing pixel values of the images in different modalities based on the first masks of the first RGB image, the first infrared image, and the first depth image to obtain a second mask; and determining a minimum bounding box of the object, and calibrating the minimum bounding box to obtain a segmentation result of the object.

## Claims

### Claim 1

An object segmentation method based on multimodal data fusion, comprising steps of:
S1: acquiring an initial RGB image, an initial infrared image, and an initial depth image that contain an object;
S2: aligning the initial RGB image, the initial infrared image, and the initial depth image to obtain a first RGB image, a first infrared image, and a first depth image in a same coordinate system respectively;
S3: specifying an initial prompt point in the first RGB image, and acquiring first masks characterizing an object region from the first RGB image, the first infrared image, and the first depth image through the initial prompt point respectively;
S4: fusing pixel values of the images in different modalities based on the first masks of the first RGB image, the first infrared image, and the first depth image to obtain a second mask; and
S5: determining a minimum bounding box of the object based on the second mask, and calibrating the minimum bounding box to obtain a segmentation result of the object.

### Claim 2 (depends on 1)

The object segmentation method based on multimodal data fusion according to claim 1, wherein a method for obtaining a first RGB image, a first infrared image, and a first depth image in a same coordinate system in S2 comprises:
S21: extracting feature points from the initial RGB image, the initial infrared image, and the initial depth image respectively, wherein each feature point has one feature descriptor, and the feature descriptor is an encoded vector that contains local information surrounding the feature point;
S22: constructing one approximate nearest neighbor search data structure for a feature descriptor set of each modality image;
S23: randomly selecting an approximate nearest neighbor search data structure of a modality image, and searching the approximate nearest neighbor search data structure based on a feature point descriptor of another modality image to obtain candidate matching points;
S24: obtaining a geometric relationship estimation matrix between any two modality images based on the candidate matching points; and
S25: aligning all the modality images based on the geometric relationship estimation matrix to obtain the first RGB image, the first infrared image, and the first depth image in the same coordinate system.

### Claim 3 (depends on 1)

The object segmentation method based on multimodal data fusion according to claim 1, wherein a method for acquiring first masks characterizing an object region from the first RGB image, the first infrared image, and the first depth image in S3 comprises:
S31: selecting one or more initial prompt points from the object region of the first RGB image; and
S32: obtaining the first mask of the first RGB image based on the initial prompt point, and mapping the coordinates of the initial prompt point into the first infrared image and the first depth image respectively to obtain the first mask of the first infrared image and the first mask of the first depth image.

### Claim 4 (depends on 1)

The object segmentation method based on multimodal data fusion according to claim 1, wherein a method for obtaining a second mask in S4 comprises:
S41: obtaining information entropy of a channel of each modality image based on the first masks of the first RGB image, the first infrared image, and the first depth image;
S42: obtaining, based on the information entropy of the channel of each modality image, a weight corresponding to the first mask of the modality image;
S43: performing weighted fusion on the first masks of the three modality images based on the weight corresponding to the first mask of the modality image to obtain a fused value of each pixel; and
S44: comparing the fused value of each pixel with an estimation threshold, and retaining pixels whose fused value is greater than the estimation threshold to obtain the second mask.

### Claim 5 (depends on 4)

The object segmentation method based on multimodal data fusion according to claim 4, wherein a calculation method for obtaining information entropy of a channel of each modality image in S41 comprises:
a calculation formula for the information entropy HRGB of the channel of the first RGB image is:
H
RGB
=
-
∑
i
=
0
2
⁢
5
⁢
5
⁢
P
RGB
(
i
RGB
)
⁢
log
⁢
P
RGB
(
i
RGB
)
,
a calculation formula for the information entropy HIR of the channel of the first infrared image is:
H
IR
=
-
∑
i
=
0
2
⁢
5
⁢
5
⁢
P
IR
(
i
IR
)
⁢
log
⁢
P
IR
(
i
IR
)
,
a calculation formula for the information entropy HDepth of the channel of the first depth image is:
H
Depth
=
-
∑
i
=
0
2
⁢
5
⁢
5
⁢
P
Depth
(
i
Depth
)
⁢
log
⁢
P
Depth
(
i
Depth
)
,
wherein PRGB(IRGB), PDepth(iDepth), and PIR(iIR) are respectively probability distributions of an RGB image, a depth image, and an infrared image on the pixel values iRGB, iDepth, and iIR.

### Claim 6 (depends on 4)

The object segmentation method based on multimodal data fusion according to claim 4, wherein a method for obtaining a weight corresponding to the first mask of the modality image in S42 is: using the reciprocal of the information entropy of the channel of each modality image as the weight corresponding to the first mask.

### Claim 7 (depends on 4)

The object segmentation method based on multimodal data fusion according to claim 4, wherein a method for obtaining a fused value F(x,y) of each pixel in S43 comprises:
F
⁡
(
x
,
y
)
=
W
RGB
·
RGB
mask
(
x
,
y
)
+
W
IR
·
IR
m
⁢
a
⁢
s
⁢
k
(
x
,
y
)
+
W
Depth
·
Depth
mask
(
x
,
y
)
W
RGB
+
W
IR
+
W
Depth
,
wherein WRGB is the weight of the channel of the first RGB image, and RGBmask(x,y) is the coordinates of any pixel in the first mask of the first RGB image; WIR is the weight of the channel of the first infrared image, IRmask(x,y) is the coordinates of any pixel in the first mask of the first infrared image; and WDepth is the weight of the channel of the first depth image, and Depthmask(x,y) is the coordinates of any pixel in the first mask of the first depth image.

### Claim 8 (depends on 4)

The object segmentation method based on multimodal data fusion according to claim 4, wherein a method for calculating the estimation threshold θ is:
θ
=
H
_
+
σ
H
,
wherein H denotes a mean value of the information entropy of the channels of the modality images,
H
_
=
H
RGB
+
H
Depth
+
H
IR
3
,
HRGB is the information entropy of the first RGB image, HDepth is the information entropy of the first depth image, and HIR is the information entropy of the first infrared image; and σH denotes a standard deviation of the information entropy of the channels of the modality images, and
σ
H
=
(
H
RGB
-
H
_
)
2
+
(
H
Depth
-
H
_
)
2
+
(
H
IR
-
H
_
)
2
3
.

### Claim 9 (depends on 1)

The object segmentation method based on multimodal data fusion according to claim 1, wherein a method for obtaining a segmentation result of the object in S5 comprises:
S51: superimposing the second mask on the initial RGB image to obtain the minimum bounding box of the object;
S52: traversing all pixels in the minimum bounding box, and converting an RGB value of each pixel to a color feature closest to the RGB value based on a color mapping table;
S53: processing the converted color features, and clustering a region in the minimum bounding box into N classes to obtain a clustering result;
S54: randomly selecting n points from a class with the largest total data amount of the clustering result as auxiliary points, and adding the auxiliary points to a prompt point set to obtain an updated prompt point set;
S55: generating a new mask based on the updated prompt point set, and calculating a change in the intersection over union between a current mask and a mask generated in a previous iteration; and
S56: determining whether the change is less than a preset threshold:
if the change is not less than the preset threshold, returning to Step S54; and
if the change is less than the preset threshold, stopping iterations, outputting a current mask, and superimposing the current mask on the initial RGB image to obtain the segmentation result of the object.

### Claim 10 (depends on 1)

An image annotation tool, comprising:
a receiving module, configured to receive at least one creation instruction input through an object interface, and when a plurality of creation instructions are input, queue the creation instructions based on priority or submission order;
an acquisition module, configured to acquire a target quantity of images to be annotated based on a resource address comprised in the creation instruction;
an annotation module, configured to automatically annotate the image to be annotated using the object segmentation method based on multimodal data fusion according to claim 1, and display an annotation result on the object interface; and
a saving module, configured to save the annotation result in multiple file formats.
