# US 2020/0402250 A1 · Unsupervised learning of image depth and ego-motion prediction neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402250 A1
- Document ID: US20200402250A1
- Status: Published application
- Kind: A1
- Application number: 17/010,967
- Filed: 2020-09-03
- Published: 2020-12-24
- Earliest claimed: 2017-11-15
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Anelia Angelova; Martin Wicke; Reza Mahjourian
- Web page: https://patentgrep.com/patent/US20200402250A1

## CPC

- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/579: from motion
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06T 7/285: using a sequence of stereo image pairs
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system includes a neural network implemented by one or more computers, in which the neural network includes an image depth prediction neural network and a camera motion estimation neural network. The neural network is configured to receive a sequence of images. The neural network is configured to process each image in the sequence of images using the image depth prediction neural network to generate, for each image, a respective depth output that characterizes a depth of the image, and to process a subset of images in the sequence of images using the camera motion estimation neural network to generate a camera motion output that characterizes the motion of a camera between the images in the subset. The image depth prediction neural network and the camera motion estimation neural network have been jointly trained using an unsupervised learning technique.

## Claims

### Claim 1

A system comprising:
a neural network implemented by one or more computers, wherein the neural network comprises an image depth prediction neural network and a camera motion estimation neural network, wherein the neural network is configured to receive a sequence of images, and
process each image in the sequence of images using the image depth prediction neural network to generate, for each image, a respective depth output that characterizes a depth of the image, and
process a subset of images in the sequence of images using the camera motion estimation neural network to generate a camera motion output that characterizes the motion of a camera between the images in the subset,
wherein the image depth prediction neural network and the camera motion estimation neural network have been jointly trained using an unsupervised learning technique.

### Claim 2 (depends on 1)

The system of claim 1, wherein the sequence of images are frames of a video captured by the camera.

### Claim 3 (depends on 1)

The system of claim 1, wherein the depth output comprises an estimated depth value for each pixel of a plurality of pixels in the image that represents a respective distance of a scene depicted at the pixel from a focal plane of the image.

### Claim 4 (depends on 1)

The system of claim 1, wherein the camera motion output is a transformation matrix that transforms the position and orientation of the camera from its point of view while taking a first image in the subset to its point of view while taking a second image in the subset.

### Claim 5 (depends on 1)

The system of claim 1, wherein the image depth prediction neural network includes convolutional neural network layers.

### Claim 6 (depends on 1)

The system of claim 1, wherein the camera motion estimation neural network includes convolutional neural network layers.

### Claim 7 (depends on 1)

The system of claim 1, wherein the subset of images includes two images in the sequence of images.

### Claim 8 (depends on 1)

The system of claim 1, wherein the subset of images includes three or more images in the sequence of images.

### Claim 9

One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving a sequence of images;
processing, using an image depth prediction neural network, each image in the sequence of images to generate, for each image, a respective depth output that characterizes a depth of the image; and
processing, using a camera motion estimation neural network, a subset of images in the sequence of images to generate a camera motion output that characterizes the motion of a camera between the images in the subset,
wherein the image depth prediction neural network and the camera motion estimation neural network have been jointly trained using an unsupervised learning technique.

### Claim 10 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the sequence of images are frames of a video captured by the camera.

### Claim 11 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the depth output comprises an estimated depth value for each pixel of a plurality of pixels in the image that represents a respective distance of a scene depicted at the pixel from a focal plane of the image.

### Claim 12 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the camera motion output is a transformation matrix that transforms the position and orientation of the camera from its point of view while taking a first image in the subset to its point of view while taking a second image in the subset.

### Claim 13 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the image depth prediction neural network includes convolutional neural network layers.

### Claim 14 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the camera motion estimation neural network includes convolutional neural network layers.

### Claim 15 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the subset of images includes two images in the sequence of images.

### Claim 16 (depends on 9)

The one or more non-transitory computer-readable storage media of claim 9, wherein the subset of images includes three or more images in the sequence of images.

### Claim 17

A computer-implemented method comprising:
receiving a sequence of images;
processing, using an image depth prediction neural network, each image in the sequence of images to generate, for each image, a respective depth output that characterizes a depth of the image; and
processing, using a camera motion estimation neural network, a subset of images in the sequence of images to generate a camera motion output that characterizes the motion of a camera between the images in the subset,
wherein the image depth prediction neural network and the camera motion estimation neural network have been jointly trained using an unsupervised learning technique.

### Claim 18 (depends on 17)

The method of claim 17, wherein the sequence of images are frames of a video captured by the camera.

### Claim 19 (depends on 17)

The method of claim 17, wherein the depth output comprises an estimated depth value for each pixel of a plurality of pixels in the image that represents a respective distance of a scene depicted at the pixel from a focal plane of the image.

### Claim 20 (depends on 17)

The method of claim 17, wherein the camera motion output is a transformation matrix that transforms the position and orientation of the camera from its point of view while taking a first image in the subset to its point of view while taking a second image in the subset.
