# US 2021/0281814 A1 · Systems and methods for self-supervised depth estimation according to an arbitrary camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0281814 A1
- Document ID: US20210281814A1
- Status: Published application
- Kind: A1
- Application number: 16/899,906
- Filed: 2020-06-12
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: Toyota Research Institute, Inc.
- Applicants: Toyota Research Institute, Inc.
- Inventors: Vitor Guizilini; Igor Vasiljevic; Rares A. Ambrus; Sudeep Pillai; Adrien David Gaidon
- Web page: https://patentgrep.com/patent/US20210281814A1

## CPC

- G06T 15/20: Perspective computation
- H04N 13/128: Adjusting depth or disparity
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06T 9/002: {using neural networks}
- G06T 15/06: Ray-tracing
- H04N 13/261: with monoscopic-to-stereoscopic image conversion
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 23/60: Control of cameras or camera modules
- G06N 3/045: Combinations of networks
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

System, methods, and other embodiments described herein relate to improving depth estimates for monocular images using a neural camera model that is independent of a camera type. In one embodiment, a method includes receiving a monocular image from a pair of training images derived from a monocular video. The method includes generating, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera having the camera type. The method includes creating a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

## Claims

### Claim 1

A depth system for improving depth estimates for monocular images using a neural camera model that is independent of a camera type, comprising:
one or more processors;
a memory communicably coupled to the one or more processors and storing:
a ray module including instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a monocular image from a pair of training images derived from a monocular video, and
generate, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera with a defined type; and
a training module including instructions that, when executed by the one or more processors, cause the one or more processors to create a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

### Claim 2 (depends on 1)

The depth system of claim 1, wherein the training module includes instructions to create the synthesized image including instructions to apply the neural camera model by:
lifting pixels to produce three-dimensional points using the ray surface, the depth map, and a camera offset, and
projecting the three-dimensional points onto a context image to create the synthesized image.

### Claim 3 (depends on 2)

The depth system of claim 2, wherein the training module includes instructions to lift the pixels including instructions to scale predicted ray vectors from the ray surface using the depth map and adjust the predicted ray vectors according to the camera offset that is an origin of a reference coordinate system.

### Claim 4 (depends on 2)

The depth system of claim 2, wherein the training module includes instructions to project including instructions to apply a softmax approximation to derive each pixel in the synthesized image by identifying a predicted ray vector from the ray surface that corresponds with a direction associated with each of the three-dimensional points as defined relative to the camera offset, and
wherein the ray module includes instructions to generate the ray surface including instructions to learn the camera type to provide the ray surface as part of the neural camera model that approximates the camera type for a set of training data including the pair of training images.

### Claim 5 (depends on 2)

The depth system of claim 2, wherein the training module includes instructions to project includes determining a patch-based data association for searching each pixel in the synthesized image by defining search grids for target pixels according to coordinates of respective ones of the target pixels and a defined grid size, and
wherein the training module includes instructions to project the three-dimensional points into the synthesized image includes applying a softmax approximation with an annealing temperature to search over the respective search grids.

### Claim 6 (depends on 1)

The depth system of claim 1, wherein the ray surface is comprised of a residual component and a fixed component, wherein the residual component is learned by the ray surface network and the fixed component is a geometric prior for the camera type,
wherein the image character associated with the camera type includes at least a format of the monocular image and lens distortion, and
wherein the ray surface associates pixels within the monocular image with directions in an environment from which light that generates the pixels in the camera originates.

### Claim 7 (depends on 1)

The depth system of claim 1, wherein the training module further includes instructions to provide the synthesized image as part of training a depth model for generating the depth estimates.

### Claim 8 (depends on 1)

The depth system of claim 1, wherein the training module includes instructions to train the depth model to produce the depth estimates according to a self-supervised structure from motion (SfM) process, and
wherein training the depth model includes using the synthesized image to derive a loss according to a loss function that updates at least the depth model, and the ray surface network.

### Claim 9

A non-transitory computer-readable medium for improving depth estimates for monocular images using a neural camera model that is independent of a camera type and including instructions that when executed by one or more processors cause the one or more processors to:
receive a monocular image from a pair of training images derived from a monocular video;
generate, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera with a defined type; and
create a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

### Claim 10 (depends on 9)

The non-transitory computer-readable medium of claim 9, wherein the instructions to create the synthesized image include instructions to apply the neural camera model by:
lifting pixels to produce three-dimensional points using the ray surface, the depth map, and a camera offset, and
projecting the three-dimensional points onto a context image to create the synthesized image.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to lift the pixels include instructions to scale predicted ray vectors from the ray surface using the depth map and adjust the predicted ray vectors according to the camera offset that is an origin of a reference coordinate system.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to project include instructions to apply a softmax approximation to derive each pixel in the synthesized image by identifying a predicted ray vector from the ray surface that corresponds with a direction associated with each of the three-dimensional points as defined relative to the camera offset, and
wherein the instructions to generate the ray surface include instructions to learn the camera type to provide the ray surface as part of the neural camera model that approximates the camera type for a set of training data including the pair of training images.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the instructions to project include determining a patch-based data association for searching each pixel in the synthesized image by defining search grids for target pixels according to coordinates of respective ones of the target pixels and a defined grid size, and
wherein the instructions to project the three-dimensional points into the synthesized image includes applying a softmax approximation with an annealing temperature to search over the respective search grids.

### Claim 14

A method of improving depth estimates for monocular images using a neural camera model that is independent of a camera type, comprising:
receiving a monocular image from a pair of training images derived from a monocular video;
generating, using a ray surface network, a ray surface that approximates an image character of the monocular image as produced by a camera having the camera type; and
creating a synthesized image according to at least the ray surface and a depth map associated with the monocular image.

### Claim 15 (depends on 14)

The method of claim 14, wherein creating the synthesized image includes applying the neural camera model by:
lifting pixels to produce three-dimensional points using the ray surface, the depth map, and a camera offset, and
projecting the three-dimensional points onto a context image to create the synthesized image.

### Claim 16 (depends on 15)

The method of claim 15, wherein lifting the pixels includes scaling predicted ray vectors from the ray surface using the depth map and adjusting the predicted ray vectors according to the camera offset that is an origin of a reference coordinate system.

### Claim 17 (depends on 15)

The method of claim 15, wherein projecting includes applying a softmax approximation to derive each pixel in the synthesized image by identifying a predicted ray vector from the ray surface that corresponds with a direction associated with each of the three-dimensional points as defined relative to the camera offset, and
wherein generating the ray surface includes learning the camera type to provide the ray surface as part of the neural camera model that approximates the camera type for a set of training data including the pair of training images.

### Claim 18 (depends on 15)

The method of claim 15, wherein projecting includes determining a patch-based data association for searching each pixel in the synthesized image by defining search grids for target pixels according to coordinates of respective ones of the target pixels and a defined grid size, and
wherein projecting the three-dimensional points into the synthesized image includes applying a softmax approximation with an annealing temperature to search over the respective search grids.

### Claim 19 (depends on 14)

The method of claim 14, wherein the ray surface is comprised of a residual component and a fixed component, wherein the residual component is learned by the ray surface network and the fixed component is a geometric prior for the camera type,
wherein the image character associated with the camera type includes at least a format of the monocular image and lens distortion, and
wherein the ray surface associates pixels within the monocular image with directions in an environment from which light that generates the pixels in the camera originates.

### Claim 20 (depends on 14)

The method of claim 14, further comprising:
training a depth model to produce the depth estimates according to a self-supervised structure from motion (SfM) process,
wherein training the depth model includes using the synthesized image to derive a loss according to a loss function that updates at least the depth model, and the ray surface network.
