# US 2022/0284696 A1 · System and method for training a model to perform semantic segmentation on low visibility images using high visibility images having a close camera view

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284696 A1
- Document ID: US20220284696A1
- Status: Published application
- Kind: A1
- Application number: 17/625,590
- Filed: 2019-07-10
- Published: 2022-09-08
- Earliest claimed: 2019-07-10
- Assignees: Toyota Motor Europe; Eth Zurich (Swiss Federal Institute of Technology Zurich)
- Applicants: Toyota Motor Europe; Eth Zurich the Swiss Federal Institute of Technology Zurich
- Inventors: Wim ABBELOOS; Christos SAKARIDIS; Luc VAN GOOL; Dengxin DAI
- Web page: https://patentgrep.com/patent/US20220284696A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 7/11: Region-based segmentation
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06T 2207/10024: Color image
- G06T 2207/20028: Bilateral filtering
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system and a method for training a semantic segmentation model includes obtaining a plurality of sets of images each having an index z for visibility, iteratively training the model. Iteratively training the model includes (a) for each z above 1, obtaining preliminary semantic segmentation labels for each image of the set of images of index z−1 by applying the model to each image of the set of images of index z−1, (b) processing each preliminary semantic segmentation labels using semantic segmentation labels obtained using the model on a selected image of index 1, and obtaining processed semantic segmentation labels, (c) training the model using the set of images of index z−1 and the associated processed semantic segmentation labels, and (d) performing steps (a) to (c) for z+1.

## Claims

### Claim 1

A method for training a model to be used for semantic segmentation of images taken under low visibility conditions, comprising:
obtaining a plurality of sets of images, each set of images being associated with an index z comprised between 1 and Z, the index z indicating a level of visibility of the images of the set of images changing from 1 to Z, wherein the model is initially trained to perform semantic segmentation under the level of visibility of level 1, and
iteratively training the model comprising:
a—for each z above 1, obtaining preliminary semantic segmentation labels for each image of the set of images of index z−1 by applying the model to each image of the set of images of index z−1,
b—processing each preliminary semantic segmentation labels using semantic segmentation labels obtained using the model on a selected image of the set of images of index 1, the selected image being selected as the image having a camera view which is closest to a camera view of the image associated with the preliminary semantic segmentation label, and obtaining processed semantic segmentation labels,
c—training the model using the set of images of index z−1 and the associated processed semantic segmentation labels, and
d—performing steps a to c for z+1.

### Claim 2 (depends on 1)

The method of claim 1, comprising a preliminary step of obtaining the semantic segmentation labels for all the images of the set of images of index 1 using the model.

### Claim 3 (depends on 1)

The method of claim 1, wherein step b comprises:
using a cross bilateral filter between the semantic segmentation labels obtained using the model on a selected image of the set of images of index 1 and the image, and
performing a fusion of an output of the cross bilateral filter with the semantic segmentation labels obtained using the model on the image.

### Claim 4 (depends on 3)

The method of claim 3, wherein the cross bilateral filter performs:
S
~
1
(
p
)
=
Σ
q
∈
𝒩
⁡
(
p
)
⁢
G
σ
s
(

q
-
p

)
⁢
G
σ
r
(

I
z
(
q
)
-
I
z
(
p
)

)
⁢
S
1
(
q
)
Σ
q
∈
𝒩
⁡
(
p
)
⁢
G
σ
s
(

q
-
p

)
⁢
G
σ
r
(

I
z
(
q
)
-
I
z
(
p
)

)
.
in which:
p and q are pixel positions,
{tilde over (S)}1(p) is an output of the cross bilateral filter for a pixel p,
(p) is a neighborhood of p,
Gσs is a spatial-domain Gaussian kernel,
Gσr a color-domain kernel,
Iz(q) and Iz(p) respectively designate a color value at pixel q and pixel p in an image from a set of index z, and
S1(q) is a semantic segmentation label at pixel q for the selected image obtained using the model.

### Claim 5 (depends on 3)

The method of claim 3, wherein performing the fusion of an output of the cross bilateral filter with the semantic segmentation labels obtained using the model on the image comprises using:
S
^
z
=
F
z
F
z
+
α
⁢
F
1
⁢
S
z
+
α
⁢
F
1
F
z
+
α
⁢
F
1
⁢
S
~
1
in which:
Ŝz is an output of a fusion function,
F1 is equal to
max
c
∈
S
~
c
1
(
p
)
and is a maximum output value of a semantic segmentation label at pixel p for the selected image for all possible classes , c designating a class having a highest output,
Fz is equal to
max
c
∈
S
~
c
z
(
p
)
and is a maximum output value of a semantic segmentation label at pixel p for the image, and
a a coefficient chosen between 0 and 1.

### Claim 6 (depends on 5)

The method of claim 5, wherein α is a function of a pixel p:
α
⁡
(
p
)
=
{
α
l
,
if
⁢
c
1
=
arg
max
c
∈
𝒞
S
~
c
1
(
p
)
∈
d
and
⁢
S
c
1
z
(
p
)
≤
η
or
⁢
c
2
=
arg
max
c
∈
𝒞
S
c
z
(
p
)
∈
d
and
⁢
S
~
c
2
1
(
p
)
≤
η
,
α
h
otherwise
.

### Claim 7 (depends on 1)

The method of claim 1, comprising obtaining an annotated set of images taken having a level of visibility of level 1 with the associated semantic segmentation labels, and training the model with the annotated set of images and the associated semantic segmentation labels.

### Claim 8 (depends on 7)

The method of claim 7, further comprising generating at least one set of generated images so that the set of generated images has a level of visibility corresponding to an index above one, and training the model using the at least one set of generated images and the semantic segmentation labels associated with the annotated set of images.

### Claim 9

A system for training a model to be used for semantic segmentation of images taken under low visibility conditions, comprising:
a module for obtaining a plurality of sets of images, each set of images being associated with an index z comprised between 1 and Z, the index z indicating a level of visibility of the images of the set of images changing from 1 to Z, wherein the model is initially trained to perform semantic segmentation under the level of visibility of level 1,
a module for iteratively training the model configured to perform:
a—for each z above 1, obtaining preliminary semantic segmentation labels for each image of the set of images of index z−1 by applying the model to each image of the set of images of index z−1,
b—processing each preliminary semantic segmentation labels using semantic segmentation labels obtained using the model on a selected image of the set of images of index 1, the selected image being selected as the image having a camera view which is closest to a camera view of the image associated with the preliminary semantic segmentation label, and obtaining processed semantic segmentation labels,
c—training the model using the set of images of index z−1 and the associated processed semantic segmentation labels, and
d—performing steps a to c for z+1.

### Claim 10 (depends on 1)

A model to be used for semantic segmentation trained by the method of claim 1.

### Claim 11 (depends on 1)

A vehicle comprising an image acquisition module and a system comprising the model trained by the method of claim 1 to perform semantic segmentation on the images acquired by the image acquisition module.