# US 2024/0202942 A1 · Image processing device determining motion vector between frames, and method thereby

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202942 A1
- Document ID: US20240202942A1
- Status: Published application
- Kind: A1
- Application number: 18/591,943
- Filed: 2024-02-29
- Published: 2024-06-20
- Earliest claimed: 2021-08-30
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Youjin LEE; Yongsup Park; Sangmi Lee; Gyehyun Kim; Beomseok Kim; Youngo Park; Taeyoung Jang; Kwangpyo Choi
- Web page: https://patentgrep.com/patent/US20240202942A1

## CPC

- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/254: involving subtraction of images
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 5/20: using local operators
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An image processing device configured to obtain difference maps between a first frame or a first feature map corresponding to the first frame, and second feature maps corresponding to a second frame, obtain third feature maps and fourth feature maps by performing pooling processes on the difference maps according to a first size and a second size, obtain modified difference maps by weighted-summing the third feature maps and the fourth feature maps, identify any one collocated sample based on sizes of sample values of collocated samples of the modified difference maps corresponding to a current sample of the first frame, and determine a filter kernel used to obtain the second feature map corresponding to the modified difference map including the identified collocated sample, as a motion vector of the current sample.

## Claims

### Claim 1

An image processing device comprising:
at least one memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions to:
obtain a plurality of difference maps between a first frame or a first feature map corresponding to the first frame, and a plurality of second feature maps corresponding to a second frame;
obtain a plurality of third feature maps and a plurality of fourth feature maps by performing a first pooling process based on a first size, and a second pooling process based on a second size, on the plurality of difference maps;
obtain a plurality of modified difference maps by weighted-summing the plurality of third feature maps and the plurality of fourth feature maps;
identify any one collocated sample based on sizes of sample values of collocated samples of the plurality of modified difference maps corresponding to a current sample of the first frame; and
determine a filter kernel used to obtain one of the plurality of second feature maps corresponding to one of the plurality of modified difference maps comprising the identified collocated sample, as a motion vector of the current sample.

### Claim 2 (depends on 1)

The image processing device of claim 1, wherein a first stride used in the first pooling process and a second stride used in the second pooling process are different from each other.

### Claim 3 (depends on 2)

The image processing device of claim 2, wherein the first size and the first stride are greater than the second size and the second stride.

### Claim 4 (depends on 3)

The image processing device of claim 3, wherein the first size and the first stride are k and k is a natural number, and
wherein the second size and the second stride are k/2.

### Claim 5 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to obtain, from a neural network, a first weight applied to the plurality of third feature maps, and a second weight applied to the plurality of fourth feature maps.

### Claim 6 (depends on 1)

The image processing device of claim 1, wherein the at least one processor is further configured to execute the one or more instructions to:
obtain the plurality of modified difference maps by weighted-summing the plurality of third feature maps and the plurality of fourth feature maps, based on a first preliminary weight and a second preliminary weight that are output from a neural network;
determine motion vectors corresponding to samples of the first frame, from the plurality of modified difference maps; and
motion-compensate the second frame based on the motion vectors, and wherein the neural network is trained based on first loss information corresponding to a difference between the motion-compensated second frame and the first frame.

### Claim 7 (depends on 6)

The image processing device of claim 6, wherein the neural network is trained further based on second loss information indicating how much a sum of the first preliminary weight and the second preliminary weight differs from a predetermined threshold.

### Claim 8 (depends on 6)

The image processing device of claim 6, wherein the neural network is trained further based on third loss information indicating how small negative values of the first preliminary weight and the second preliminary weight are.

### Claim 9 (depends on 1)

The image processing device of claim 1, wherein each of the first pooling process and the second pooling process comprises an average pooling process or a median pooling process.

### Claim 10 (depends on 1)

The image processing device of claim 1, wherein the first feature map is obtained through first convolution processing on the first frame based on a first filter kernel, and
wherein the plurality of second feature maps are obtained through second convolution processing on the second frame based on a plurality of second filter kernels.

### Claim 11 (depends on 10)

The image processing device of claim 10, wherein a first distance between samples of the first frame on which a first convolution operation with the first filter kernel is performed, and a second distance between samples of the second frame on which a second convolution operation with the plurality of second filter kernels is performed, are greater than 1.

### Claim 12 (depends on 10)

The image processing device of claim 10, wherein, in the first filter kernel, a sample corresponding to the current sample of the first frame has a preset first value, and other samples of the first filter kernel have a value of 0.

### Claim 13 (depends on 12)

The image processing device of claim 12, wherein, in the plurality of second filter kernels, any one sample has a preset second value, and other samples of the plurality of second filter kernels have a value of 0, and
wherein positions of samples having the preset second value in the plurality of second filter kernels are different from each other.

### Claim 14 (depends on 13)

The image processing device of claim 13, wherein a sign of the preset first value and a sign of the preset second value are opposite to each other.

### Claim 15

An image processing method performed by an image processing device, the image processing method comprising:
obtaining a plurality of difference maps between a first frame or a first feature map corresponding to the first frame, and a plurality of second feature maps corresponding to a second frame;
obtaining a plurality of third feature maps and a plurality of fourth feature maps by performing a first pooling process based on a first size, and a second pooling process based on a second size, on the plurality of difference maps;
obtaining a plurality of modified difference maps by weighted-summing the plurality of third feature maps and the plurality of fourth feature maps;
identifying any one collocated sample by considering sizes of sample values of collocated samples of the plurality of modified difference maps corresponding to a current sample of the first frame; and
determining a filter kernel used to obtain one of the plurality of second feature maps corresponding to one of the plurality of modified difference maps comprising the identified collocated sample, as a motion vector of the current sample.

### Claim 16 (depends on 15)

The image processing method of claim 15, wherein a first stride used in the first pooling process, and a second stride used in the second pooling process, are different from each other.

### Claim 17 (depends on 16)

The image processing method of claim 16, wherein the first size and the first stride are greater than the second size and the second stride.

### Claim 18 (depends on 17)

The image processing method of claim 17, wherein the first size and the first stride are k and k is a natural number, and
wherein the second size and the second stride are k/2.

### Claim 19 (depends on 15)

The image processing method of claim 15, further comprising:
obtaining, from a neural network, a first weight applied to the plurality of third feature maps, and a second weight applied to the plurality of fourth feature maps.

### Claim 20 (depends on 15)

The image processing method of claim 15, further comprising:
obtaining the plurality of modified difference maps by weighted-summing the plurality of third feature maps and the plurality of fourth feature maps, based on a first preliminary weight and a second preliminary weight that are output from a neural network;
determining motion vectors corresponding to samples of the first frame, from the plurality of modified difference maps; and
motion-compensating the second frame based on the motion vectors,
wherein the neural network is trained based on first loss information corresponding to a difference between the motion-compensated second frame and the first frame.
