# US 2025/0166339 A1 · Semi-supervised and robust multispectral video semantic segmentation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166339 A1
- Document ID: US20250166339A1
- Status: Published application
- Kind: A1
- Application number: 18/955,003
- Filed: 2024-11-21
- Published: 2025-05-22
- Earliest claimed: 2023-11-22
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Wenbo LI; Yilin SHEN; Hongxia JIN
- Web page: https://patentgrep.com/patent/US20250166339A1

## CPC

- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 7/11: Region-based segmentation
- G06V 10/30: Noise filtering
- G06V 10/58: relating to hyperspectral data
- G06V 10/82: using neural networks
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes generating a pair of features of a current frame by inputting the current frame into a first cross-collaborative consistency learning (C3L) model, the current frame comprising a red-green-blue (RGB) image and a thermal image; generating a pair of denoised features by inputting the of pair of features of the current frame and one or more pairs of features of past frames into a denoised memory read (DMR) model; generating an updated pair of denoised features by inputting the pair of denoised features into a second C3L model, the updated pair of denoised features comprising an updated RGB image feature and an updated thermal feature; and generating a segmentation mask by inputting the updated pair of features into a segmentation head.

## Claims

### Claim 1

A method performed by at least one processor, the method comprising:
generating a pair of features of a current frame by inputting the current frame into a first cross-collaborative consistency learning (C3L) model, the current frame comprising a red-green-blue (RGB) image and a thermal image;
generating a pair of denoised features by inputting the of pair of features of the current frame and one or more pairs of features of past frames into a denoised memory read (DMR) model;
generating an updated pair of denoised features by inputting the pair of denoised features into a second C3L model, the updated pair of denoised features comprising an updated RGB image feature and an updated thermal feature; and
generating a segmentation mask by inputting the updated pair of features into a segmentation head.

### Claim 2 (depends on 1)

The method according to claim 1, wherein at least one of the RGB image and the thermal image is labeled.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the RGB image and the thermal image are not labeled.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the first C3L model comprises
a first semantic segmentation neural network that receives the RGB image and outputs an RGB feature of the pair of features,
a second semantic segmentation neural network that receives the thermal image and outputs a thermal image feature of the pair of features,
a first convolutional network that receives an output of the first semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the RGB image, and
a second convolutional network that receives an output of the second semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the thermal image.

### Claim 5 (depends on 4)

The method according to claim 4, further comprising:
determining a cross modality consistency between the RGB image and the thermal image based on a loss function that uses the probabilistic segmentation prediction of the modality of the RGB image, the probabilistic segmentation prediction of the modality of the thermal image, a pseudo label for the modality of the RGB image, and a pseudo label for the modality of the thermal image.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising:
generating the one or more pairs of features of the past frames by inputting the past frames into a third C3L model, each past frame comprising a past RGB image and a past thermal image;
storing the one or more pairs of features in a denoised memory bank in accordance with a reliability condition;
inputting the one or more pairs of features stored in the denoised memory bank into the DMR model; and
wherein each past frame is unlabeled.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the second C3L model comprises:
a first convolutional network that receives the updated RGB image feature and outputs a probabilistic segmentation prediction of a modality of the RGB image, and
a second convolutional network that receives the updated thermal image feature and outputs a probabilistic segmentation prediction of a modality of the thermal image.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising:
determining a cross modality consistency between the updated RGB image feature and the updated thermal image feature based on a loss function that uses the probabilistic segmentation prediction of the modality of the RGB image, the probabilistic segmentation prediction of the modality of the thermal image, a pseudo label for the modality of the RGB image, and a pseudo label for the modality of the thermal image.

### Claim 9

An apparatus comprising:
a memory storing one or more instructions; and
a processor operatively coupled to the memory and configured to execute the one or more instructions stored in the memory,
wherein the one or more instructions, when executed by the processor, cause the apparatus to:
generate a pair of features of a current frame by inputting the current frame into a first cross-collaborative consistency learning (C3L) model, the current frame comprising a red-green-blue (RGB) image and a thermal image,
generate a pair of denoised features by inputting the of pair of features of the current frame and one or more pairs of features of past frames into a denoised memory read (DMR) model,
generate an updated pair of denoised features by inputting the pair of denoised features into a second C3L model, the updated pair of denoised features comprising an updated RGB image feature and an updated thermal feature, and
generate a segmentation mask by inputting the updated pair of features into a segmentation head.

### Claim 10 (depends on 9)

The apparatus according to claim 9, wherein at least one of the RGB image and the thermal image is labeled.

### Claim 11 (depends on 9)

The apparatus according to claim 9, wherein the RGB image and the thermal image are not labeled.

### Claim 12 (depends on 9)

The apparatus according to claim 9, wherein the first C3L model comprises:
a first semantic segmentation neural network that receives the RGB image and outputs an RGB feature of the pair of features,
a second semantic segmentation neural network that receives the thermal image and outputs a thermal image feature of the pair of features,
a first convolutional network that receives an output of the first semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the RGB image, and
a second convolutional network that receives an output of the second semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the thermal image.

### Claim 13 (depends on 9)

The apparatus according to claim 9, wherein the one or more instructions, when executed by the processor, cause the apparatus to:
determine a cross modality consistency between the RGB image and the thermal image based on a loss function that uses the probabilistic segmentation prediction of the modality of the RGB image, the probabilistic segmentation prediction of the modality of the thermal image, a pseudo label for the modality of the RGB image, and a pseudo label for the modality of the thermal image.

### Claim 14 (depends on 9)

The apparatus according to claim 9, wherein the one or more instructions, when executed by the processor, cause the apparatus to:
generate the one or more pairs of features of the past frames by inputting the past frames into a third C3L model, each past frame comprising a past RGB image and a past thermal image,
storing the one or more pairs of features in a denoised memory bank in accordance with a reliability condition,
inputting the one or more pairs of features stored in the denoised memory bank into the DMR model, and
wherein each past frame is unlabeled.

### Claim 15 (depends on 9)

The apparatus according to claim 9, wherein the second C3L model comprises:
a first convolutional network that receives the updated RGB image feature and outputs a probabilistic segmentation prediction of a modality of the RGB image, and
a second convolutional network that receives the updated thermal image feature and outputs a probabilistic segmentation prediction of a modality of the thermal image.

### Claim 16 (depends on 9)

The apparatus according to claim 9, wherein the one or more instructions, when executed by the processor, cause the apparatus to:
determine a cross modality consistency between the updated RGB image feature and the updated thermal image feature based on a loss function that uses the probabilistic segmentation prediction of the modality of the RGB image, the probabilistic segmentation prediction of the modality of the thermal image, a pseudo label for the modality of the RGB image, and a pseudo label for the modality of the thermal image.

### Claim 17

A non-transitory computer readable medium having instructions stored therein, which when executed by a processor cause the processor to perform a method comprising:
generating a pair of features of a current frame by inputting the current frame into a first cross-collaborative consistency learning (C3L) model, the current frame comprising a red-green-blue (RGB) image and a thermal image;
generating a pair of denoised features by inputting the of pair of features of the current frame and one or more pairs of features of past frames into a denoised memory read (DMR) model;
generating an updated pair of denoised features by inputting the pair of denoised features into a second C3L model, the updated pair of denoised features comprising an updated RGB image feature and an updated thermal feature; and
generating a segmentation mask by inputting the updated pair of features into a segmentation head.

### Claim 18 (depends on 17)

The non-transitory computer readable medium according to claim 17, wherein at least one of the RGB image and the thermal image is labeled.

### Claim 19 (depends on 17)

The non-transitory computer readable medium according to claim 17, wherein the RGB image and the thermal image are not labeled.

### Claim 20 (depends on 17)

The non-transitory computer readable medium according to claim 17, wherein the first C3L model comprises:
a first semantic segmentation neural network that receives the RGB image and outputs an RGB feature of the pair of features,
a second semantic segmentation neural network that receives the thermal image and outputs a thermal image feature of the pair of features,
a first convolutional network that receives an output of the first semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the RGB image, and
a second convolutional network that receives an output of the second semantic segmentation neural network and outputs a probabilistic segmentation prediction of a modality of the thermal image.
