# US 2017/0256033 A1 · Image Upsampling using Global and Local Constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0256033 A1
- Document ID: US20170256033A1
- Status: Published application
- Kind: A1
- Application number: 15/059,629
- Filed: 2016-03-03
- Published: 2017-09-07
- Earliest claimed: 2016-03-03
- Assignees: Mitsubishi Electric Research Laboratories, Inc.
- Applicants: Mitsubishi Electric Research Laboratories, Inc.
- Inventors: Oncel Tuzel; Yuichi Taguchi
- Web page: https://patentgrep.com/patent/US20170256033A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 3/4046: using neural networks
- G06V 40/166: {using acquisition arrangements}
- H04N 19/136: Incoming video signal characteristics or properties
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20024: Filtering details
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20172: Image enhancement details
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method upsamples an image using a non-linear fully connected neural network to produce only global details of an upsampled image and interpolates the image to produce a smooth upsampled image. The method concatenates the global details and the smooth upsampled image into a tensor and applies a sequence of nonlinear convolutions to the tensor using a convolutional neural network to produce the upsampled image.

## Claims

### Claim 1

A method for upsampling an image, comprising:
upsampling the image using a non-linear fully connected neural network to produce only global details of an upsampled image;
interpolating the image to produce a smooth upsampled image;
concatenating the global details and the smooth upsampled image into a tensor; and
applying a sequence of nonlinear convolutions to the tensor using a convolutional neural network to produce the upsampled image, wherein steps of the method are performed by a processor.

### Claim 2 (depends on 1)

The method of claim 1, wherein the fully connected network, the interpolation, and the convolution are concurrently trained to reduce an error between upsampled set of images and corresponding set of high-resolution images.

### Claim 3 (depends on 2)

The method of claim 2, wherein the fully connected network is a neural network, and wherein the training produces weights for each neuron of the neural network.

### Claim 4 (depends on 2)

The method of claim 2, wherein the interpolation uses different weights for interpolating different pixels of the image, and wherein the training produces the different weights of the interpolation.

### Claim 5 (depends on 2)

The method of claim 2, wherein the training produces weights for each neuron of the sequence of nonlinear convolutions.

### Claim 6 (depends on 5)

The method of claim 5, further comprising:
padding each nonlinear convolution in the sequence to the resolution of the of the upsampled image.

### Claim 7 (depends on 1)

The method of claim 1, wherein the global details of the upsampled image encode high frequency details of the upsampled image to enforce global constraints on the upsampled image, and wherein the smooth upsampled image encodes low frequency details of the upsampled image to enforce local constraints on the upsampled image.

### Claim 8 (depends on 7)

The method of claim 7, wherein the sequence of nonlinear convolutions enforces data constraints on the upsampled image.

### Claim 9 (depends on 1)

The method of claim 1, wherein the image is a low resolution image of a face, and wherein the upsampled image is a high resolution image of the face, wherein the high resolution is greater than the low resolution.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
receiving the image from an imaging device; and
rendering the upsampled image onto a display device.

### Claim 11 (depends on 10)

The method of claim 10, wherein the imaging device includes one or combination of a camera, a computer, a scanner, a mobile device, and a webcam, and wherein the display device includes one or combination of a computer monitor, the camera, a television, a projector, or a mobile device.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
filtering an original image to produce the image.

### Claim 13

A computer system for upsampling an image including a processor and a memory, wherein the instructions stored in the memory configure the processor to
upsample the image using a non-linear fully connected neural network to produce only global details of an upsampled image;
interpolate the image to produce a smooth upsampled image;
concatenate the global details and the smooth upsampled image into a tensor; and
apply a sequence of nonlinear convolutions to the tensor using a convolutional neural network to produce the upsampled image.

### Claim 14 (depends on 13)

The system of claim 13, wherein the fully connected network, the interpolation, and the convolution are concurrently trained to reduce an error between upsampled set of images and corresponding set of high resolution images.

### Claim 15 (depends on 14)

The system of claim 14, wherein the fully connected network is a neural network, and wherein the training produces weights for each neuron of the neural network, wherein the interpolation uses different weights for interpolating different pixels of the image, and wherein the training produces the different weights of the interpolation, and wherein the training produces weights for each neuron of the sequence of nonlinear convolutions.

### Claim 16 (depends on 13)

The system of claim 13, wherein the global details of the upsampled image encode high frequency details of the upsampled image to enforce global constraints on the upsampled image, wherein the smooth upsampled image encodes low frequency details of the upsampled image to enforce local constraints on the upsampled image, and wherein the sequence of nonlinear convolutions enforces data constraints on the upsampled image.

### Claim 17 (depends on 13)

The system of claim 13, wherein the image is a low resolution image of a face, and wherein the upsampled image is a high resolution image of the face, wherein the high resolution is greater than the low resolution.

### Claim 18 (depends on 13)

The system of claim 13, further comprising:
an imaging device to determine the image; and
a display device to display the upsampled image.

### Claim 19 (depends on 16)

The system of claim 16, wherein the imaging device includes one or combination of a camera, a computer, a scanner, a mobile device, and a webcam, and wherein the display device includes one or combination of a computer monitor, the camera, a television, a projector, or a mobile device.

### Claim 20 (depends on 13)

The system of claim 13, further comprising:
a filter for filtering an original image to produce the image.
