# US 2019/0244331 A1 · Unsupervised Learning Approach for Video Deblurring

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244331 A1
- Document ID: US20190244331A1
- Status: Published application
- Kind: A1
- Application number: 15/887,907
- Filed: 2018-02-02
- Published: 2019-08-08
- Earliest claimed: 2018-02-02
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corp.
- Inventors: Jinwei Gu; Orazio Gallo; Ming-Yu Liu; Jan Kautz; Huaijin Chen
- Web page: https://patentgrep.com/patent/US20190244331A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/73: Deblurring; Sharpening
- G06V 30/19173: {Classification techniques}
- G06N 3/047: Probabilistic or stochastic networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing method extracts consecutive input blurry frames from a video, and generates sharp frames corresponding to the input blurry frames. An optical flow is determined between the sharp frames, and the optical flow is used to compute a per-pixel blur kernel. The blur kernel is used to reblur each of the sharp frames into a corresponding re-blurred frame. The re-blurred frame is used to fine-tune the deblur network by minimizing the distance between the re-blurred frame and the input blurry frame.

## Claims

### Claim 1

An image processing method comprising:
extracting a plurality of consecutive input blurry frames from a video;
generating a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurry frames;
determining an optical flow between the sharp frames;
applying the optical flow to compute a per-pixel blur kernel; and
applying the blur kernel to reblur each of the sharp frames into a corresponding re-blurred frame.

### Claim 2 (depends on 1)

The image processing method of claim 1, wherein:
generating the sharp frames is performed with a first neural network; and
generating the optical flow is performed with a second neural network.

### Claim 3 (depends on 2)

The image processing method of claim 2, wherein the first neural network is trained using the re-blurred frames.

### Claim 4 (depends on 3)

The image processing method of claim 3, wherein the training of the first neural network minimizes a distance between each of the input blurry frames and a corresponding re-blurred frame.

### Claim 5 (depends on 3)

The image processing method of claim 3, wherein the training of the first neural network comprises backpropagation of weights derived from both of a determined supervised loss in the re-blurred frames and a determined unsupervised loss in the re-blurred frames.

### Claim 6 (depends on 2)

The image processing method of claim 2, wherein the first neural network is one of DVD and DeblurGAN.

### Claim 7 (depends on 2)

The image processing method of claim 2, wherein the second neural network is a FlowNetS network.

### Claim 8 (depends on 1)

The image processing method of claim 1, wherein the reblur is performed utilizing a precomputed lookup table that maps the optical flow to the blur kernel using bilinear interpolation.

### Claim 9 (depends on 1)

The image processing method of claim 1, wherein the reblur is performed using a per-pixel convolution with the blur kernel.

### Claim 10 (depends on 9)

The image processing method of claim 9, wherein the blur kernel comprises a pair of piecewise linear line segments per pixel.

### Claim 11

An image processing apparatus comprising:
a graphics processing unit; and
logic to configure the graphics processing unit to:
extract a plurality of consecutive input blurry frames from a video;
generate a plurality of sharp frames, each of the sharp frames corresponding to one of the input blurry frames;
determine an optical flow between the sharp frames;
apply the optical flow to compute a per-pixel blur kernel; and
apply the blur kernel to reblur each of the sharp frames into a corresponding re-blurred frame.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the logic further configures the graphics processing unit to:
generate the sharp frames with a first neural network; and
generate the optical flow with a second neural network.

### Claim 13 (depends on 12)

The apparatus of claim 12, further comprising logic to train the first neural network using the re-blurred frames.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the training of the first neural network minimizes a distance between each of the input blurry frames and a corresponding re-blurred frame.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the training of the first neural network comprises backpropagation of weights derived from both of a determined supervised loss in the re-blurred frames and a determined unsupervised loss in the re-blurred frames.

### Claim 16 (depends on 12)

The apparatus of claim 12, wherein the first neural network is one of DVD and DeblurGAN.

### Claim 17 (depends on 12)

The apparatus of claim 12, wherein the second neural network is a FlowNetS network.

### Claim 18 (depends on 11)

The apparatus of claim 11, wherein the reblur is performed utilizing a precomputed lookup table that maps the optical flow to the blur kernel using bilinear interpolation.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein the reblur is performed using a per-pixel convolution with the blur kernel.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the blur kernel comprises a pair of piecewise linear line segments per pixel.