# US 2021/0233210 A1 · Method and system of real-time super-resolution image processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233210 A1
- Document ID: US20210233210A1
- Status: Published application
- Kind: A1
- Application number: 17/213,489
- Filed: 2021-03-26
- Published: 2021-07-29
- Earliest claimed: 2021-03-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Noam Elron; Alexander Itskovich; Shahar S Yuval; Noam Levy
- Web page: https://patentgrep.com/patent/US20210233210A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, system, and article is directed to real-time super-resolution image processing using neural networks.

## Claims

### Claim 1

A computer-implemented method of super-resolution image processing, comprising:
obtaining at least one lower resolution (LR) image;
generating at least one high resolution (HR) image comprising inputting image data of the at least one LR image into at least one super-resolution (SR) neural network;
separately generating weights or biases or both comprising using statistics of the LR image; and
providing the weights or biases or both to the SR neural network to be used to generate the at least one HR image.

### Claim 2 (depends on 1)

The method of claim 1 wherein the weights or biases are generated at a control flow operated in parallel to a pixel processing flow that operates the SR neural network.

### Claim 3 (depends on 2)

The method of claim 2 wherein a rate of generating the weights or biases or both at the control flow is slower than a frame rate at the SR neural network.

### Claim 4 (depends on 1)

The method of claim 1 comprising obtaining a video sequence of the LR images, and wherein the weights or biases or both are updated at predetermined image intervals along the video sequence rather than with every image in the video sequence.

### Claim 5 (depends on 1)

The method of claim 1 wherein the weights or biases or both are generated to be used for a target image and are arranged to be used on images near the target image along a video sequence of the images when the weights or biases are generated too late to be used on the target image.

### Claim 6 (depends on 1)

The method of claim 1 comprising inputting at least one of the LR images into a convolutional neural network (CNN) to generate statistical values.

### Claim 7 (depends on 6)

The method of claim 6 wherein the statistics comprises global weighted average values each representing an entire image.

### Claim 8 (depends on 1)

The method of claim 1 wherein the weights or biases are generated by using the statistics to generate a convolution kernel of weights.

### Claim 9 (depends on 1)

The method of claim 1 wherein generating the weights or biases comprises inputting user preference values and the statistics into a weight engine neural network to generate a proportioning coefficient to apply to a convolution kernel of weights.

### Claim 10

A system for image processing, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and storing a video sequence of low resolution (LR) images, the at least one processor being configured to operate by:
generating higher resolution (HR) images comprising inputting image data of the LR images into at least one super-resolution (SR) neural network;
separately generating weights or biases or both comprising using LR image statistics and generated at intervals of a number of images along the video sequence; and
providing the weights or biases or both to the SR neural network at the intervals to be used to generate the HR images.

### Claim 11 (depends on 10)

The system of claim 10 wherein the statistics are generated in a statistics neural network that receives the image data as input and generates weights from a weight layer to be applied to statistics values of a value layer to generate outputted statistics.

### Claim 12 (depends on 10)

The system of claim 10 wherein the statistics comprises a vector of global weighted values, each value representing an entire image.

### Claim 13 (depends on 10)

The system of claim 10 wherein the generating of weights or biases comprises inputting the statistics into a weight engine neural network that generates a proportion coefficient to be applied to a convolution kernel of weights.

### Claim 14 (depends on 10)

The system of claim 10 wherein the neural network uses the latest available weights or bias or both rather than waiting for updated weights or bias or both to be generated when the generation of the weights or bias or both is late.

### Claim 15 (depends on 10)

The system of claim 10 wherein the SR neural network is trained by intentionally blurring input training images.

### Claim 16

A method of image processing comprising:
obtaining initial training images;
generating blurred training images comprising intentionally blurring one or more of the initial training images;
generating low resolution training images or high resolution training images or both and comprising using the blurred images; and
training a super-resolution neural network to generate high resolution pixel values comprising using the low resolution training images or high resolution training images or both.

### Claim 17 (depends on 16)

The method of claim 16 comprising blurring the initial training images comprising changing image data of the initial training images to simulate blurring caused at least in part by real camera lenses.

### Claim 18 (depends on 16)

The method of claim 16 comprising blurring the initial training images comprising applying a point spread function (PSF)-related value to image data of the initial training images.

### Claim 19 (depends on 18)

The method of claim 18 comprising randomly selecting a PSF-related value to apply to individual initial training images and among multiple PSF-related values available to be used for blurring and each related to a different PSF value.

### Claim 20 (depends on 16)

The method of claim 16 comprising training the SR neural network to convert the resolution on input blurred LR images caused by various PSF values.

### Claim 21 (depends on 16)

The method of claim 16 comprising decimating a blurred image to generate a desired size of the low resolution training image or high resolution training image or both.

### Claim 22

At least one non-transitory article having at least one computer readable medium comprising a plurality of instructions that in response to being executed on a computing device, cause the computing device to operate by:
obtaining initial training images;
generating blurred training images comprising intentionally blurring one or more of the initial training images;
generating low resolution training images or high resolution training images or both and comprising using the blurred images; and
training a super-resolution neural network to generate high resolution pixel values comprising using the low resolution training images or high resolution training images or both.

### Claim 23 (depends on 22)

The article of claim 22 wherein the instructions cause the computing device to operate by generating a stretch factor stretching a size of a point spread function (PSF) and set to compensate for reducing a size of the initial training image.

### Claim 24 (depends on 23)

The article of claim 23 wherein the stretch factor is used to convolve the initial training image to form the blurred images.

### Claim 25 (depends on 22)

The article of claim 22 wherein the instructions cause the computing device to operate during run-time by inputting low resolution images to the SR neural network while separately generating weights or biases or both of the SR neural network based on statistics of the low resolution images.
