# US 2022/0092796 A1 · Image Processing Method and Device Thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0092796 A1
- Document ID: US20220092796A1
- Status: Published application
- Kind: A1
- Application number: 17/424,811
- Filed: 2019-02-21
- Published: 2022-03-24
- Earliest claimed: 2019-02-21
- Assignees: Korea Advanced Institute of Science and Technology
- Applicants: Korea Advanced Institute of Science and Technology
- Inventors: Junyong Noh; Julie Alfonsine E. Lelong; Bumki Kim; Hanui Lee; Gukho Kim; Mohammad Reza Karimi Dastjerdi; Allen Kim; Jiwon Lee
- Web page: https://patentgrep.com/patent/US20220092796A1

## CPC

- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 2207/10016: Video; Image sequence
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An image processing device according to one embodiment estimates first optical flow information of a pixel unit on the basis of consecutive frames, and estimates a polynomial model corresponding to the optical flow information. The image processing device estimates second optical flow information of holes included in the frames on the basis of the polynomial model, and inpaints an input image on the basis of the first optical flow information and/or the second optical flow information.

## Claims

### Claim 1

An image processing method, comprising:
receiving an input image of sequential frames;
estimating first optical flow information of a pixel unit based on the frames;
estimating a polynomial model corresponding to the first optical flow information;
estimating second optical flow information of a hole in the frames based on the polynomial model; and
inpainting the input image based on at least one of the first optical flow information and the second optical flow information.

### Claim 2 (depends on 1)

The image processing method of claim 1, wherein the estimating of the polynomial model comprises:
estimating a correlation coefficient in the polynomial model based on a motion pattern corresponding to a coordinate system of the input image and the first optical flow information.

### Claim 3 (depends on 2)

The image processing method of claim 2, wherein the polynomial model is modeled as a product of a matrix corresponding to the motion pattern and a vector corresponding to the correlation coefficient.

### Claim 4 (depends on 3)

The image processing method of claim 3, wherein the estimating of the correlation coefficient comprises:
estimating the correlation coefficient such that a difference between a value of the polynomial model and a value of the first optical flow information is minimized.

### Claim 5 (depends on 1)

The image processing method of claim 1, wherein the inpainting comprises:
backtracking a value of a pixel corresponding to the hole based on at least one of the first optical flow information and the second optical flow information.

### Claim 6 (depends on 1)

The image processing method of claim 1, wherein the hole corresponds to a region removed from the input image.

### Claim 7 (depends on 1)

The image processing method of claim 1, further comprising:
extracting feature points from the frames; and
removing a rotation between the frames by matching the feature points between the frames.

### Claim 8 (depends on 1)

The image processing method of claim 1, further comprising:
generating inpainting mask data indicating the hole based on the frames,
wherein the inpainting comprises:
inpainting the input image based further on the inpainting mask data.

### Claim 9 (depends on 1)

The image processing method of claim 1, wherein the estimating of the first optical flow information comprises:
estimating a vector indicating an optical flow between a pixel in one frame of the frames and pixels in remaining frames excluding the one frame, the vector corresponding to the pixel in the one frame.

### Claim 10 (depends on 1)

The image processing method of claim 1, wherein the input image comprises 360-degree (°) panorama video data.

### Claim 11 (depends on 1)

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image processing method of claim 1.

### Claim 12

An image processing device, comprising:
a receiver configured to receive an input image of sequential frames;
an estimator configured to estimate first optical flow information of a pixel unit based on the frames, estimate a polynomial model corresponding to the first optical flow information, and estimate second optical flow information of a hole in the frames based on the polynomial model; and
an inpainter configured to inpaint the input image based on at least one of the first optical flow information and the second optical flow information.

### Claim 13 (depends on 12)

The image processing device of claim 12, wherein the estimator is configured to:
estimate the polynomial model by estimating a correlation coefficient in the polynomial model based on a motion pattern corresponding to a coordinate system of the input image and the first optical flow information.

### Claim 14 (depends on 13)

The image processing device of claim 13, wherein the polynomial model is modeled as a product of a matrix corresponding to the motion pattern and a vector corresponding to the correlation coefficient.

### Claim 15 (depends on 12)

The image processing device of claim 12, wherein the inpainter is configured to:
backtrack a value of a pixel corresponding to the hole based on at least one of the first optical flow information and the second optical flow information.

### Claim 16 (depends on 12)

The image processing device of claim 12, wherein the hole corresponds to a region removed from the input image.

### Claim 17 (depends on 12)

The image processing device of claim 12, further comprising:
a preprocessor configured to remove a rotation between the frames by extracting feature points from the frames and matching the feature points between the frames.

### Claim 18 (depends on 12)

The image processing device of claim 12, wherein the inpainter is configured to:
inpaint the input image based further on inpainting mask data indicating the hole.

### Claim 19 (depends on 12)

The image processing device of claim 12, wherein the estimator is configured to:
estimate the first optical flow information by estimating a vector indicating an optical flow between a pixel in one frame of the frames and pixels of remaining frames excluding the one frame, the vector corresponding to the pixel in the one frame.

### Claim 20 (depends on 12)

The image processing device of claim 12, wherein the input image comprises 360-degree (°) panorama video data.
