# US 2021/0241473 A1 · System for image compositing including training with synthetic data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241473 A1
- Document ID: US20210241473A1
- Status: Published application
- Kind: A1
- Application number: 17/081,843
- Filed: 2020-10-27
- Published: 2021-08-05
- Earliest claimed: 2020-01-30
- Assignees: Weta Digital Limited
- Applicants: Weta Digital Limited
- Inventors: Tobias B. Schmidt; Erik B. Edlund; Dejan Momcilovic; Josh Hardgrave
- Web page: https://patentgrep.com/patent/US20210241473A1

## CPC

- G06T 7/33: using feature-based methods
- G06T 7/50: Depth or shape recovery
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Embodiments allow live action images from an image capture device to be composited with computer generated images in real-time or near real-time. The two types of images (live action and computer generated) are composited accurately by using a depth map. In an embodiment, the depth map includes a “depth value” for each pixel in the live action image. In an embodiment, steps of one or more of feature extraction, matching, filtering or refinement can be implemented, at least in part, with an artificial intelligence (AI) computing approach using a deep neural network with training. A combination of computer-generated (“synthetic”) and live-action (“recorded”) training data is created and used to train the network so that it can improve the accuracy or usefulness of a depth map so that compositing can be improved.

## Claims

### Claim 1

A tangible processor-readable medium including instructions executable by one or more processors for:
selecting a deep neural network that has been trained using a dataset derived, at least in part, from synthetic data;
using the selected deep neural network to process image information from one or more auxiliary cameras to generate a depth map; and
correlating the depth map with at least a portion of picture elements in at least one picture image obtained from an image capture device; and
using the correlated depth map to composite one or more digital elements with one or more picture elements.

### Claim 2 (depends on 1)

The tangible processor-readable medium of claim 1, wherein the dataset includes recorded data.

### Claim 3 (depends on 1)

The tangible processor-readable medium of claim 1, wherein the dataset includes third party recorded data.

### Claim 4 (depends on 1)

The tangible processor-readable medium of claim 1, the one or more tangible media further comprising logic for:
re-projecting the disparity map into an image from the image capture device.

### Claim 5 (depends on 4)

The tangible processor-readable medium of claim 4, the one or more tangible media further comprising logic for:
infilling holes in the re-projected disparity map.

### Claim 6 (depends on 1)

The tangible processor-readable medium of claim 1, further comprising:
a signal interface for receiving image camera information and providing the image camera information to the one or more processors for processing by the deep neural network.

### Claim 7 (depends on 6)

The tangible processor-readable medium of claim 6, wherein the image camera information includes a focal length of the image camera.

### Claim 8 (depends on 7)

The tangible processor-readable medium of claim 7, wherein the operation is in real time.

### Claim 9

A method for determining picture element depths, the method comprising:
selecting a deep neural network that has been trained using a dataset derived, at least in part, from synthetic data derived from a scene;
using the selected deep neural network to process image information of the scene from one or more auxiliary cameras to generate a depth map; and
correlating the depth map with at least a portion of picture elements in at least one of the picture images using the correlated depth map to composite one or more digital elements with one or more picture elements.

### Claim 10 (depends on 9)

The method of claim 9, wherein the dataset includes recorded data.

### Claim 11 (depends on 9)

The method of claim 9, wherein the dataset includes third party recorded data.

### Claim 12 (depends on 9)

The method of claim 9, further comprising:
re-projecting the disparity map into an image from the image capture device.

### Claim 13 (depends on 12)

The method of claim 12, further comprising infilling holes in the re-projected disparity map.

### Claim 14 (depends on 13)

The method of claim 13, further comprising receiving image camera information and providing the deep neural network for processing.

### Claim 15 (depends on 14)

The method of claim 14, wherein the image camera information includes a focal length of the image camera.

### Claim 16 (depends on 15)

The method of any one of claim 15, wherein the operation is in real time.

### Claim 17

A method for generating a depth map, the method comprising:
receiving a dataset including a plurality of images and depths of objects in a scene;
using the dataset to train a deep neural network to assist, at least in part, in generating a depth map for use in real-time compositing of a live action recording taking place in the scene.

### Claim 18 (depends on 17)

The method of claim 17, wherein the dataset includes recorded data.

### Claim 19 (depends on 17)

The method of claim 17, wherein the dataset includes third party recorded data.

### Claim 20

(canceled)
