US 10,855,966 B2Granted

View interpolation of multi-camera array images with flow estimation and image super resolution using deep learning

AssigneeIntel Corporation

InventorsBindita Chaudhuri, Fan Zhang, Oscar Nestares

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques related to interpolating an intermediate view image from multi-view images are discussed. Such techniques include downsampling first and second images that represent a view of a scene, generating a disparity map based on applying a first CNN to the downscaled first and second images, translating the downscaled first and second images using the disparity map, applying a second CNN to the translated downscaled first and second images and the disparity map to generate a downscaled intermediate image, and upscaling the downscaled intermediate image to an intermediate image at the resolution of the first and second images using an image super-resolution convolutional neural network.

Claims20

Dependent claims

A system for implementing a convolutional neural network (CNN) comprising:

a memory to store first and second images, wherein the first and second images comprise different views of a scene and are at a first resolution;
and
one or more processors coupled to the memory, the one or more processors to:
generate at least one disparity map based at least in part on application of a first convolutional neural network to a first input volume comprising first and second downscaled images corresponding to the first and second images, respectively;
determine first and second translated downscaled images based at least in part on the disparity map;
apply a second convolutional neural network to a second input volume comprising the first and second translated downscaled images and the disparity map to generate a downscaled intermediate image comprising a view between the first and second translated downscaled images;
and
generate an intermediate image at the first resolution based at least in part on application of an image super-resolution convolutional neural network to the downscaled intermediate image.

5 more elements

A computer-implemented method for generating an intermediate image from multi-view images comprising:

generating at least one disparity map based at least in part on applying a first convolutional neural network to a first input volume comprising first and second downscaled images corresponding to first and second images that are at a first resolution that comprise different views of a scene;
determining first and second translated downscaled images based at least in part on the disparity map;
applying a second convolutional neural network to a second input volume comprising the first and second translated downscaled images and the disparity map to generate a downscaled intermediate image comprising a view between the first and second translated downscaled images;
and
generating the intermediate image at the first resolution based at least in part on applying an image super-resolution convolutional neural network to the downscaled intermediate image.

2 more elements

At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to generate an intermediate image from multi-view images by:

generating at least one disparity map based at least in part on applying a first convolutional neural network to a first input volume comprising first and second downscaled images corresponding to first and second images that are at a first resolution that comprise different views of a scene;
determining first and second translated downscaled images based at least in part on the disparity map;
applying a second convolutional neural network to a second input volume comprising the first and second translated downscaled images and the disparity map to generate a downscaled intermediate image comprising a view between the first and second translated downscaled images;
and
generating the intermediate image at the first resolution based at least in part on applying an image super-resolution convolutional neural network to the downscaled intermediate image.

2 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/752,346Filed 24 Jan 2020 · continuation
Priority: 25 Sep 2018
Published: US 2020/0162715 A121 May 2020
Granted: 1 Dec 2020
Term ends: Not estimatedThe source records don’t support an estimate.

Classification22 codes

H04N 13/111
Processing image signals: transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
G06T 1/20
General purpose image data processing: processor architectures; Processor configuration, e.g. pipelining
G06T 3/20
Geometric image transformations in the plane of the image: linear translation of whole images or parts thereof, e.g. panning
G06T 3/4053
Scaling of whole images or parts thereof, e.g. expanding or contracting: based on super-resolution, i.e. the output image resolution being higher…

Family

Application16/141,426Parent of this continuation · filed 25 Sep 2018
US 2020/0162715 A1Pre-grant publication · published 21 May 2020
US 10,547,823 B2Parent of this continuation
US 2019/0045168 A1Published application
US 10,855,966 B2This patent

From the US filing history. Foreign family members aren’t listed.