# US 2021/0334543 A1 · Method for semantic segmentation based on knowledge distillation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334543 A1
- Document ID: US20210334543A1
- Status: Published application
- Kind: A1
- Application number: 17/222,055
- Filed: 2021-04-05
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: Ajou University Industry-Academic Cooperation Foundation
- Applicants: Ajou University Industry-Academic Cooperation Foundation
- Inventors: Yong Seok HEO; Sangyong PARK
- Web page: https://patentgrep.com/patent/US20210334543A1

## CPC

- G06T 7/11: Region-based segmentation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure provides a knowledge distillation based semantic image segmentation method including inputting an input image to a teacher network and a student network; normalizing a first feature vector corresponding to each pixel in a feature map of a last layer of the teacher network and normalizing a second feature vector corresponding to each pixel in a feature map of a last layer of the student network; generating the first channel and space association matrix and the second channel and space association matrix based on the normalized first feature vector and the normalized second feature vector, and defining a first loss function based on an Euclidean norm value of the difference between the first channel and space association matrix and the second channel and space association matrix.

## Claims

### Claim 1

A knowledge distillation based semantic image segmentation method, comprising:
inputting an input image to a teacher network and a student network;
normalizing a first feature vector corresponding to each pixel in a feature map of a last layer of the teacher network and normalizing a second feature vector corresponding to each pixel in a feature map of a last layer of the student network;
generating a first channel and space association matrix and a second channel and space association matrix based on the normalized first feature vector and the normalized second feature vector; and
defining a first loss function based on an Euclidean norm value of the difference between the first channel and space association matrix and the second channel and space association matrix.

### Claim 2 (depends on 1)

The knowledge distillation based semantic image segmentation method according to claim 1, further comprising:
defining an entire loss function using the first loss function and a cross entropy loss function, and
determining one label among a plurality of labels for each pixel of an input image using the entire loss function.

### Claim 3 (depends on 2)

The knowledge distillation based semantic image segmentation method according to claim 2, wherein the defining of an entire loss function includes:
defining the entire loss function based on the first loss function, a weight for the first loss function, the cross entropy loss function, and a weight for the cross entropy loss function.

### Claim 4 (depends on 2)

The knowledge distillation based semantic image segmentation method according to claim 2, further comprising:
training the student network using the entire loss function.

### Claim 5 (depends on 1)

The knowledge distillation based semantic image segmentation method according to claim 1, wherein when the number of labels for the input image is C, the generating of a first channel and space association matrix includes:
calculating a first channel associated vector with respect to the C labels, based on the normalized first feature vector and a circularly shifted vector of the normalized first feature vector;
calculating a first channel association matrix by concatenating and realigning the first channel associated vector for the C labels; and
determining a correlation matrix of the first channel association matrix as a first channel and space association matrix.

### Claim 6 (depends on 5)

The knowledge distillation based semantic image segmentation method according to claim 5, wherein the generating of a second channel and space association matrix includes:
calculating a second channel associated vector with respect to the C labels, based on the normalized second feature vector and a circularly shifted vector of the normalized second feature vector;
calculating a second channel association matrix by concatenating and realigning the second channel associated vector for the C labels; and
determining a correlation matrix of the second channel association matrix as a second channel and space association matrix.

### Claim 7 (depends on 5)

The knowledge distillation based semantic image segmentation method according to claim 5, wherein the first channel association vector is determined by performing element-wise multiplication on the normalized first feature vector and a circularly shifted vector of the normalized first feature vector with respect to C labels.

### Claim 8 (depends on 5)

The knowledge distillation based semantic image segmentation method according to claim 5, wherein a width of the first channel association matrix and the second channel association matrix is a value obtained by multiplying a width and a height of a feature map of a last layer of the teacher network and a height of the first channel association matrix and the second channel association matrix is a square of C.

### Claim 9 (depends on 1)

The knowledge distillation based semantic image segmentation method according to claim 1, wherein the widths and the heights of the first channel and space association matrix and the second channel and space association matrix are values obtained by multiplying the width and the height of the feature map of the last layer of the teacher network.
