# US 2019/0356905 A1 · Self-supervised training of a depth estimation system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0356905 A1
- Document ID: US20190356905A1
- Status: Published application
- Kind: A1
- Application number: 16/413,907
- Filed: 2019-05-16
- Published: 2019-11-21
- Earliest claimed: 2018-05-17
- Assignees: Niantic, Inc.
- Applicants: Niantic, Inc.
- Inventors: Clément Godard; Oisin Mac Aodha; Michael Firman; Gabriel J. Brostow
- Web page: https://patentgrep.com/patent/US20190356905A1

## CPC

- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- G06T 7/579: from motion
- G06T 7/593: from stereo images
- G06T 7/73: using feature-based methods
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30244: Camera pose
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N 2013/0088: {Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for training a depth estimation model and methods for use thereof are described. Images are acquired and input into a depth model to extract a depth map for each of the plurality of images based on parameters of the depth model. The method includes inputting the images into a pose decoder to extract a pose for each image. The method includes generating a plurality of synthetic frames based on the depth map and the pose for each image. The method includes calculating a loss value with an input scale occlusion and motion aware loss function based on a comparison of the synthetic frames and the images. The method includes adjusting the plurality of parameters of the depth model based on the loss value. The trained model can receive an image of a scene and generate a depth map of the scene according to the image.

## Claims

### Claim 1

A depth estimation system stored on a computer readable storage medium, wherein the depth estimation is manufactured by a process comprising:
acquiring a set of images;
inputting the images into a depth model to extract depth maps for the images based on parameters of the depth model;
inputting the images into a pose decoder to extract poses for the images;
generating synthetic frames based on the depth map and the poses for the images;
calculating a loss value with an input scale occlusion and motion aware loss function based on a comparison of the synthetic frames and the images; and
adjusting the parameters of the depth model based on the loss value.

### Claim 2 (depends on 1)

The depth estimation system of claim 1, wherein the set of images comprises monocular video with each of image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:
for images of the monocular video, generating synthetic frames at adjacent timestamps.

### Claim 3 (depends on 2)

The depth estimation system of claim 2, wherein calculating the loss value with the loss function comprises:
calculating, for images of the monocular video, differentials between generated synthetic frames and image with matching timestamps.

### Claim 4 (depends on 3)

The depth estimation system of claim 3, wherein:
the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,
a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and
calculating the loss value with the loss function further comprises:
calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and
for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.

### Claim 5 (depends on 3)

The depth estimation system of claim 3, wherein calculating the loss value with the loss function further comprises:
identifying one or more static features in the monocular video,
wherein the loss value is based on filtering out the differentials of the one or more static features.

### Claim 6 (depends on 5)

The depth estimation system of claim 5, wherein identifying one or more static features in the monocular video comprises:
identifying a region in a first image at a first timestamp of the monocular video and in a second image at a second timestamp of the monocular video;
determining whether the object is in a similar position between the first image and the second image; and
in response to determining that the object is in a similar position between the first image and the second image, defining the region as a static feature.

### Claim 7 (depends on 1)

The depth estimation system of claim 1, wherein the set of images are at a first resolution and the extracted depth features are at a second resolution that is lower than the first resolution, the manufacturing process further comprising:
upsampling the extracted depth map into the first resolution from the second resolution.

### Claim 8 (depends on 1)

The depth estimation system of claim 1, wherein the set of images comprises stereoscopic image pairs with each stereoscopic image pair captured by a stereoscopic pair of cameras, wherein generating the synthetic frames comprises:
for each stereoscopic image pair, generating a synthetic frame based on a first image of the stereoscopic image pair.

### Claim 9 (depends on 8)

The depth estimation system of claim 8, wherein adjusting the parameters of the model based on the comparison of the synthetic frames and the set of images comprises:
calculating, for each stereoscopic image pair, a differential between the generated synthetic frame and a second image of the stereoscopic image pair; and
adjusting the parameters to minimize the differentials.

### Claim 10

A computer-implemented method comprising:
receiving an image of a scene;
inputting the image into a trained depth model, the depth model trained with a process including:
acquiring a set of images;
inputting the images into the depth model to extract depth maps for the images based on parameters of the depth model;
inputting the images into a pose decoder to extract poses for the images;
generating synthetic frames based on the depth maps and the poses for the images;
calculating a loss value with an input scale occlusion and motion aware loss function based on a comparison of the synthetic frames and the images; and
adjusting the parameters of the depth model based on a comparison of the synthetic frames and the images; and
generating, by the trained model, a depth map of the scene corresponding to the image of the scene.

### Claim 11 (depends on 10)

The method of claim 10, wherein the set of images comprises monocular video with each of image of the monocular video captured at a corresponding timestamp, wherein generating the synthetic frames comprises:
for images of the monocular video, generating synthetic frames at adjacent timestamps.

### Claim 12 (depends on 11)

The method of claim 11, wherein calculating the loss value with the loss function comprises:
calculating, for images of the monocular video, differentials between generated synthetic frames and image with matching timestamps.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the set of images includes a first image at a first timestamp, a second image at a second timestamp, and a third image at a third timestamp,
a first synthetic frame is generated at the second timestamp with the first image and a second synthetic frame is generated at the second timestamp with the third image, and
calculating the loss value with the loss function further comprises:
calculating a first set of differentials between the first synthetic frame and the second image and a second set of differentials between the second synthetic frame and the second image; and
for each pixel of the second image, identifying a minimum differential between the first set of differentials and the second set of differentials, wherein the loss value is based on the minimum differentials.

### Claim 14 (depends on 11)

The method of claim 11, wherein calculating the loss value with the loss function further comprises:
identifying one or more static features in the monocular video with, wherein the loss value is based on filtering out the differentials of the one or more static features.

### Claim 15 (depends on 14)

The method of claim 14, wherein identifying one or more static features in the monocular video comprises:
identifying a region in a first image at a first timestamp of the monocular video and in a second image at a second timestamp of the monocular video;
determining whether the object is in a similar position between the first image and the second image; and
in response to determining that the object is in a similar position between the first image and the second image, defining the region as a static feature.

### Claim 16 (depends on 10)

The method of claim 10, wherein the set of images are at a first resolution and the extracted depth features are at a second resolution that is lower than the first resolution, the process for training the depth model further comprising:
upsampling the extracted depth map into the first resolution from the second resolution.

### Claim 17 (depends on 10)

The method of claim 10, wherein the set of images comprises stereoscopic image pairs with each stereoscopic image pair captured by a stereoscopic pair of cameras, wherein generating the synthetic frames comprises:
for each stereoscopic image pair, generating a synthetic frame based on a first image of the stereoscopic image pair.

### Claim 18 (depends on 17)

The method of claim 17, wherein adjusting the parameters of the model based on the comparison of the synthetic frames and the set of images comprises:
calculating, for each stereoscopic image pair, a differential between the generated synthetic frame and a second image of the stereoscopic image pair; and
adjusting the parameters to minimize the differentials.

### Claim 19 (depends on 10)

The method of claim 10, further comprising:
displaying the image of the scene augmented with virtual content based on the depth map of the scene.

### Claim 20

A non-transitory computer-readable storage medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
receiving an image of a scene;
inputting the image into a trained depth model, the depth model trained with a process including:
acquiring a set of images;
inputting the images into a depth encoder to extract abstract depth features for the images based on parameters of the depth encoder;
concatenating the abstract depth features for the images;
inputting the concatenated abstract depth features into a pose decoder to extract poses for the images;
generating synthetic frames based on parameters of the model and the poses for the images; and
adjusting the parameters of the model based on a comparison of the synthetic frames and the images; and
generating, by the trained model, a depth map of the scene corresponding to the image of the scene.
