# US 2021/0241474 A1 · System for image compositing including training with custom synthetic data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241474 A1
- Document ID: US20210241474A1
- Status: Published application
- Kind: A1
- Application number: 17/133,429
- Filed: 2020-12-23
- Published: 2021-08-05
- Earliest claimed: 2020-01-30
- Assignees: Weta Digital Limited
- Applicants: Weta Digital Limited
- Inventors: Tobias B. Schmidt; Erik B. Edlund; Dejan Momcilovic; Josh Hardgrave
- Web page: https://patentgrep.com/patent/US20210241474A1

## CPC

- G06T 7/33: using feature-based methods
- G06T 7/50: Depth or shape recovery
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Embodiments allow live action images from an image capture device to be composited with computer generated images in real-time or near real-time. The two types of images (live action and computer generated) are composited accurately by using a depth map. In an embodiment, the depth map includes a “depth value” for each pixel in the live action image. In an embodiment, steps of one or more of feature extraction, matching, filtering or refinement can be implemented, at least in part, with an artificial intelligence (AI) computing approach using a deep neural network with training. A combination of computer-generated (“synthetic”) and live-action (“recorded”) training data is created and used to train the network so that it can improve the accuracy or usefulness of a depth map so that compositing can be improved.

## Claims

### Claim 1

A method for compositing image elements, the method comprising:
obtaining custom recorded data of an aspect of an environment;
modifying the aspect in a computer program to create a rendering of a modified environment;
generating custom synthetic data from the modified environment;
selecting a deep neural network that has been trained using a dataset derived, at least in part, from the custom synthetic data;
using the selected deep neural network to process image information from one or more auxiliary cameras to generate a depth map of the environment;
correlating the depth map with at least a portion of picture elements in at least one picture image of the environment obtained from an image capture device; and
using the correlated depth map to composite one or more digital elements with one or more picture elements.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
re-projecting a disparity map into an image from the image capture device; and
infilling holes in the re-projected disparity map.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
a signal interface for receiving image camera information and providing the image camera information to one or more processors for processing by the deep neural network.

### Claim 4 (depends on 3)

The method of claim 3, wherein the image camera information includes a focal length of an image camera.

### Claim 5

An apparatus for compositing image elements, the apparatus comprising:
one or more digital processors;
a tangible processor-readable medium including instructions for:
obtaining custom recorded data of an aspect of an environment;
modifying the aspect in a computer program to create a rendering of a modified environment;
generating custom synthetic data from the modified environment;
selecting a deep neural network that has been trained using a dataset derived, at least in part, from the custom synthetic data;
using the selected deep neural network to process image information from one or more auxiliary cameras to generate a depth map of the environment;
correlating the depth map with at least a portion of picture elements in at least one picture image of the environment obtained from an image capture device; and
using the correlated depth map to composite one or more digital elements with one or more picture elements.

### Claim 6 (depends on 5)

The apparatus of claim 5, further comprising:
re-projecting a disparity map into an image from the image capture device; and
infilling holes in the re-projected disparity map.

### Claim 7 (depends on 5)

The apparatus of claim 5, further comprising:
a signal interface for receiving image camera information and providing the image camera information to one or more processors for processing by the deep neural network.

### Claim 8 (depends on 7)

The method of claim 7, wherein the image camera information includes a focal length of an image camera.

### Claim 9

A tangible processor-readable medium including instructions executable by one or more digital processors for compositing image elements, the processor-readable medium comprising one or more instructions for:
receiving custom recorded data of an aspect of an environment;
modifying the aspect in a computer program to create a rendering of a modified environment;
generating custom synthetic data from the modified environment; and
creating a training dataset derived, at least in part, from the custom synthetic data, wherein the training dataset is suitable for training a deep neural network to process image information from one or more auxiliary cameras to generate a depth map of the environment.

### Claim 10 (depends on 9)

The tangible processor-readable medium of claim 9, further comprising:
re-projecting a disparity map into an image from the image capture device.

### Claim 11 (depends on 10)

The tangible processor-readable medium of claim 10, further comprising:
infilling holes in the re-projected disparity map.

### Claim 12 (depends on 9)

The tangible processor-readable medium of claim 9, further comprising:
a signal interface for receiving image camera information and providing the image camera information to one or more processors for processing by the deep neural network.

### Claim 13 (depends on 12)

The tangible processor-readable medium of claim 12, wherein the image camera information includes a focal length of an image camera.

### Claim 14 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes an aspect includes at least a portion of a movie set.

### Claim 15 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes at least a portion of a costume.

### Claim 16 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes at least a portion of an actor.

### Claim 17 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes lighting.

### Claim 18 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes an object's position.

### Claim 19 (depends on 18)

The tangible processor-readable medium of claim 18, wherein the object includes a camera.

### Claim 20 (depends on 9)

The tangible processor-readable medium of claim 9, wherein an aspect includes a camera setting.