# US 11,115,645 B2 · Generating novel views of a three-dimensional object based on a single two-dimensional image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,115,645 B2
- Document ID: US11115645B2
- Status: Granted
- Kind: B2
- Application number: 16/230,872
- Filed: 2018-12-21
- Published: 2019-04-25
- Granted: 2021-09-07
- Earliest claimed: 2017-02-15
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Jimei Yang; Duygu Ceylan Aksit; Mehmet Ersin Yumer; Eunbyung Park
- Web page: https://patentgrep.com/patent/US11115645B2

## CPC

- G06V 10/82: using neural networks
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 15/205: {Image-based rendering}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/275: from three-dimensional [3D] object models, e.g. computer-generated stereoscopic image signals
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Embodiments are directed towards providing a target view, from a target viewpoint, of a 3D object. A source image, from a source viewpoint and including a common portion of the object, is encoded in 2D data. An intermediate image that includes an intermediate view of the object is generated based on the data. The intermediate view is from the target viewpoint and includes the common portion of the object and a disoccluded portion of the object not visible in the source image. The intermediate image includes a common region and a disoccluded region corresponding to the disoccluded portion of the object. The disoccluded region is updated to include a visual representation of a prediction of the disoccluded portion of the object. The prediction is based on a trained image completion model. The target view is based on the common region and the updated disoccluded region of the intermediate image.

## Claims

### Claim 1

A computer-readable storage medium having instructions stored thereon, which, when executed by a processor of a computing device cause the computing device to perform actions comprising:
receiving a source image that includes a first view of an object from a first viewpoint and includes a first portion of the object;
hallucinating a second portion of the object occluded in the first view of the object by generating values for masked pixels in an intermediate image of the object from a second viewpoint, the masked pixels corresponding to the second portion of the object occluded in the first view of the object;
generating a target image that includes a second view of the object from the second viewpoint, wherein the second view includes at least a sub-portion of the first portion of the object and the second portion of the object occluded in the first view, the second portion being predicted based on the generated values for the masked pixels; and
providing the target image.

### Claim 2 (depends on 1)

The one or more computer-readable storage media of claim 1, wherein the hallucinating further comprises:
generating the intermediate image that includes an intermediate view of the object from the second viewpoint, wherein the intermediate image includes a first region corresponding to the first portion of the object and a second region corresponding to the second portion of the object occluded in the first view; and
updating the second region of the intermediate image corresponding to the second portion of the object to include a prediction of the second portion of the obj ect.

### Claim 3 (depends on 2)

The one or more computer-readable storage media of claim 2, wherein the hallucinating further comprises:
generating a flow field that maps a region of the source image corresponding to the first portion of the object to the first region of the intermediate image based on a rotational transformation from the first viewpoint to the second viewpoint;
determining a pixel value for each of a plurality of pixels included in the first region of the intermediate image based at least in part on the flow field; and
generating the intermediate image further based on the pixel value for each of the plurality of pixels included in the first region of the intermediate image.

### Claim 4 (depends on 3)

The one or more computer-readable storage media of claim 3, wherein determining the pixel value comprises the pixel value for each of the plurality of pixels is based on the flow field, a sampling kernel, and a portion of data that corresponds to the first portion of the object.

### Claim 5 (depends on 2)

The one or more computer-readable storage media of claim 2, wherein generating the intermediate image is based on a geometric flow model that determines a rotation of the object about an axis that includes a component within a plane of two-dimensional (2D) data.

### Claim 6 (depends on 2)

The one or more computer-readable storage media of claim 2, wherein generating the intermediate image further comprises:
employing a geometric flow model to determine the first region of the first view of the object, wherein the first region of the first view includes the first portion of the object;
employing the geometric flow model to generate a flow field that maps the first region of the first view of the target to the first region of the second view of the target; and
generating the rotation of the first portion of the object based on the flow field, a sampling kernel, and a portion of the 2D data that corresponds to the first portion of the first view.

### Claim 7 (depends on 1)

The one or more computer-readable storage media of claim 1, wherein the hallucinating further comprises:
generating a background map based on an intersection of a foreground region of the source image that corresponds to the object and a foreground region of an intermediate image that corresponds to the object, wherein the intermediate image includes an intermediate view of the object from the second viewpoint;
determining a background region of the intermediate image based on the background map; and
determining a pixel value for each of a plurality of pixels included in the background region of the intermediate image based on a background region of the source image that is a complement of the foreground region of the source image.

### Claim 8 (depends on 1)

The one or more computer-readable storage media of claim 1, further comprising training an image completion model, the training comprising:
providing an incomplete version of a first image, wherein the first image is a two-dimensional (2D) image based on a three-dimensional (3D) training data set;
generating a predicted version of the first image based on the image completion model and the incomplete version of the first image; and
updating the image completion model based on an adversarial loss metric.

### Claim 9 (depends on 1)

The one or more computer-readable storage media of claim 1, further comprising training an image completion model, the training further comprising:
providing an incomplete version of a first image, wherein the first image is a two-dimensional (2D) image based on a three-dimensional (3D) training data set;
generating a predicted version of the first image based on the image completion model and the incomplete version of the first image;
determining a feature loss metric based on a comparison between the predicted version of the first image and the first image; and
updating the image completion model based on the feature loss metric.

### Claim 10

A method for generating views of an object, comprising:
determining a new region of a new view of an object that includes a new portion of the object, the new region based on a rotational transformation applied to two-dimensional (2D) data;
predicting the new portion of the object by generating a prediction of values for masked pixels in an intermediate image of the object from the new view, the masked pixels corresponding to the new portion of the object occluded in an initial view of the object; and
providing a target image that includes the new view of the object, wherein the new region of the new view includes the prediction of the new portion of the object.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
determining a initial region of the new view of the object, wherein the initial region includes an initial portion of the object that is disoccluded in each of the initial view and the new view of the object;
generating a rotation of the initial portion of the object, wherein the rotation includes a visual representation of the rotational transformation applied to a portion of the 2D data corresponding to the initial portion of the object included in the initial view of the object; and
providing the new view of the object, wherein the initial region of the new view includes the rotation of the initial portion of the object.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
determining an initial region of the initial view of the object that includes the initial portion of the object;
generating a flow field that maps the initial region of the initial view of the target image to the initial region of the new view of the target image; and
generating the rotation of the initial portion of the object based on the flow field, a sampling kernel, and a portion of the 2D data that corresponds to the initial portion of the initial view.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
generating a visibility map based on the rotational transformation applied to the 2D data; and
generating an intermediate view of the object, the intermediate view of the object from the new viewpoint, wherein the new region of the new view of the object is masked from the intermediate view based on the visibility map.

### Claim 14 (depends on 10)

The method of claim 10, further comprising:
generating three-dimensional (3D) coordinates for the object based on the 2D data;
determining a background region of the initial view of the object based on a background mask for the object and the 2D data, wherein the background mask is generated for the object based on the 3D coordinates for the object;
generating a background region of the new view of the object based on the background region of the initial view and the background mask for the object; and
including the background region of the new view in the provided new view of the object.

### Claim 15 (depends on 10)

The method of claim 10, further comprising:
training an image completion model, the training comprising:
providing an incomplete version of a first image, wherein the first image is a two-dimensional (2D) image based on a three-dimensional (3D) training data set;
generating a predicted version of the first image based on the image completion model and the incomplete version of the first image; and
updating the image completion model based on an adversarial loss metric.

### Claim 16 (depends on 10)

The method of claim 10, further comprising:
training an image completion model, the training comprising:
providing an incomplete version of a first image, wherein the first image is a 2D image based on the 3D training data set;
generating a predicted version of the first image based on the image completion model and the incomplete version of the first image;
determining a feature loss metric based on a comparison between the predicted version of the first image and the first image; and
updating the image completion model based on the feature loss metric.

### Claim 17 (depends on 10)

The method of claim 10, further comprising:
training an image completion model, wherein the image completion model is trained with a feature loss metric in combination with an adversarial loss metric, wherein the adversarial loss metric discriminates between synthetic images and natural images and the feature loss metric discriminates between physical features included in a plurality of image.

### Claim 18

A computing system for providing views of an object, comprising:
means for generating an image completion model based on a 3D training dataset and a geometric flow model;
means for generating a predicted image of a disoccluded portion of a 3D object depicted in a two-dimension (2D) image by hallucinating the disoccluded portion of the 3D object using generated values for masked pixels in an intermediate image of the 3D object from a second view different than an initial view, the masked pixels corresponding to the second portion of the object occluded in the initial view of the object, wherein the predicted image is generated using the image completion model and a generated intermediate view related to the intermediate image; and
means for generating a target image that includes the second view of the 3D object, wherein the target image includes the predicted image.

### Claim 19 (depends on 18)

The computing system of claim 18, further comprising:
means for training the geometric flow model.

### Claim 20 (depends on 18)

The computing system of claim 18, further comprising:
means for providing the target image to a user.