# US 2022/0284613 A1 · Generating depth images utilizing a machine-learning model built from mixed digital image sources and multiple loss function sets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284613 A1
- Document ID: US20220284613A1
- Status: Published application
- Kind: A1
- Application number: 17/186,436
- Filed: 2021-02-26
- Published: 2022-09-08
- Earliest claimed: 2021-02-26
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Wei Yin; Jianming Zhang; Oliver Wang; Simon Niklaus; Mai Long; Su Chen
- Web page: https://patentgrep.com/patent/US20220284613A1

## CPC

- G06T 7/50: Depth or shape recovery
- G06T 7/13: Edge detection
- G06T 7/143: involving probabilistic approaches, e.g. Markov random field [MRF] modelling
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/593: from stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This disclosure describes one or more implementations of a depth prediction system that generates accurate depth images from single input digital images. In one or more implementations, the depth prediction system enforces different sets of loss functions across mix-data sources to generate a multi-branch architecture depth prediction model. For instance, in one or more implementations, the depth prediction model utilizes different data sources having different granularities of ground truth depth data to robustly train a depth prediction model. Further, given the different ground truth depth data granularities from the different data sources, the depth prediction model enforces different combinations of loss functions including an image-level normalized regression loss function and/or a pair-wise normal loss among other loss functions.

## Claims

### Claim 1

In a digital medium environment for editing digital images, a computer-implemented method of generating image depth maps, the method comprising building a depth prediction machine-learning model by:
generating, utilizing the depth prediction machine-learning model, a first set of predicted depth images from a first set of ground truth digital images having a first source and a second set of predicted depth images from a second set of ground truth digital images having a second source;
determining a first measure of loss from the first set of predicted depth images utilizing a first loss function;
determining a second measure of loss from the second set of predicted depth images utilizing a second loss function that differs from the first loss function; and
tuning the depth prediction machine-learning model utilizing the first measure of loss and the second measure of loss.

### Claim 2 (depends on 1)

The method claim 1, further comprising:
generating the first set of predicted depth images by generating a set of depth maps via a first output path of the depth prediction machine-learning model; and
generating the second set of predicted depth images by generating a set of inverse depth maps via a second output path of the depth prediction machine-learning model.

### Claim 3 (depends on 2)

The method claim 2, wherein the first set of ground truth digital images comprises a first set of digital images and a set of ground truth depth maps, and wherein the second set of ground truth digital images comprises a second set of digital images and a set of ground truth inverse depth maps; and
further comprising:
determining the first measure of loss by comparing the set of depth maps to the set of ground truth depth maps from the first set of ground truth digital images utilizing the first loss function; and
determining the second measure of loss by comparing the set of inverse depth maps to the set of ground truth inverse depth maps from the second set of ground truth digital images utilizing the second loss function.

### Claim 4 (depends on 1)

The method of claim 1, further comprising generating the first measure of loss utilizing the first loss function by utilizing an image-level normalized regression loss function or a pair-wise normal loss function.

### Claim 5 (depends on 4)

The method of claim 4, further comprising generating the second measure of loss utilizing a set of loss functions comprising the image-level normalized regression loss function, the pair-wise normal loss function, and a multi-scale gradient loss function.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first set of ground truth digital images having the first source comprises LiDAR depth data having scale and shift measurements; and
further comprising determining a pair-wise normal loss from the first set of predicted depth images by utilizing a pair-wise normal loss function to compare depth data in the first set of predicted depth images to the LiDAR depth data from the ground truth depth images of the first set of ground truth digital images.

### Claim 7 (depends on 1)

The method of claim 1, wherein the second set of ground truth digital images having the second source comprises calibrated stereo-image-based depth data; and
further comprising determining an image-level normalized regression loss from the second set of predicted depth images by utilizing an image-level normalized regression loss function to compare depth data in the second set of predicted depth images to the stereo-image-based depth data from the ground truth depth images of the second set of ground truth digital images.

### Claim 8 (depends on 1)

The method of claim 1, further comprising building the depth prediction machine-learning model by:
generating, utilizing the depth prediction machine-learning model, a third set of predicted depth images from a third set of ground truth digital images having a third source, wherein the third set of ground truth digital images comprises uncalibrated stereo-image-based depth data generated at the third source;
determining a third measure of loss from the third set of predicted depth images utilizing a third loss function that determines a structure-guided ranking loss by comparing the third set of predicted depth images to the third set of ground truth digital images; and
tuning the depth prediction machine-learning model utilizing the third measure of loss.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
identifying a query digital image; and
generating a depth map from the query digital image utilizing the depth prediction machine-learning model.

### Claim 10

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computing device to build a depth prediction machine-learning model utilizing an image-level normalized regression loss by:
normalizing a ground truth depth image to generate a normalized ground truth depth image;
trimming the normalized ground truth depth image by removing outlier pixels from the normalized ground truth depth image to generate a trimmed ground truth depth image;
generating a predicted depth image utilizing the depth prediction machine-learning model from a digital image corresponding to the ground truth depth image;
determining the image-level normalized regression loss by generating a pixel-wise error between the trimmed ground truth depth image and the predicted depth image; and
tuning the depth prediction machine-learning model utilizing the image-level normalized regression loss.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the normalized ground truth depth image by utilizing a hyperbolic tangent function.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the trimmed ground truth depth image by removing outlier pixels from the normalized ground truth depth image utilizing statistical-based trimming.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
receive a first set of ground truth digital images having a first source and a second set of ground truth digital images having a second source;
detect depth range differences between the first set of ground truth digital images and the second set of ground truth digital images;
normalize the first set of ground truth digital images to a target numerical range; and
normalize the second set of ground truth digital images to the target numerical range.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, further comprising instructions that, when executed by the at least one processor, cause the computing device to:
utilize the depth prediction machine-learning model to determine a first measure of loss from the normalized first set of ground truth digital images utilizing a first set of loss functions comprising the image-level normalized regression loss; and
utilize the depth prediction machine-learning model to determine a second measure of loss from the normalized second set of ground truth digital images utilizing a second set of loss functions that is different from the first set of loss functions.

### Claim 15

A system for generating multi-class decision tree machine-learning models, the system comprising:
one or more memory devices comprising a set of ground truth digital images having a digital image and a ground truth depth image;
at least one server device configured to cause the system to build a depth prediction machine-learning model utilizing pair-wise normal loss by:
generating a predicted depth image from the digital image utilizing the depth prediction machine-learning model;
aligning the predicted depth image with the ground truth depth image;
determining the pair-wise normal loss by comparing normals from the predicted depth image to normals from the ground truth depth image; and
tuning the depth prediction machine-learning model utilizing the pair-wise normal loss.

### Claim 16 (depends on 15)

The system of claim 15, wherein the at least one server device is further configured to cause the system to generate the normals for the ground truth depth image from a three-dimensional point cloud of the ground truth depth image utilizing least-square fitting.

### Claim 17 (depends on 16)

The system of claim 16, wherein the at least one server device is further configured to cause the system to:
determine a planar region based on normals of the predicted depth image or normals of the ground truth depth image;
identify a first pair of normals from the predicted depth image within the planar region and a second pair of normals from the ground truth depth image within the planar region; and
determine the pair-wise normal loss by comparing the first pair of normals and the second pair of normals.

### Claim 18 (depends on 16)

The system of claim 16, wherein the at least one server device is further configured to cause the system to:
identify an edge based on normals of the predicted depth image or normal of the ground truth depth image;
sampling a first pair of normals from the predicted depth image based on the edge and a second pair of normals from the ground truth depth image based on the edge;
determine a first angle between the first pair of normals from the predicted depth image and a second angle between the second pair of normals from the ground truth depth image; and
determine the pair-wise normal loss by comparing the first angle and the second angle.

### Claim 19 (depends on 15)

The system of claim 15, wherein the at least one server device is further configured to cause the system to:
receive a first set of ground truth digital images having a first source comprising LiDAR depth data having scale and shift measurements; and
utilize the depth prediction machine-learning model to determine the pair-wise normal loss for planar regions and edges in the first set of ground truth digital images.

### Claim 20 (depends on 15)

The system of claim 15, wherein the at least one server device is further configured to cause the system to build the depth prediction machine-learning model utilizing the pair-wise normal loss, an image-level normalized regression loss, and a multi-scale gradient loss.
