# US 2021/0279894 A1 · Depth and motion estimations in machine learning environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279894 A1
- Document ID: US20210279894A1
- Status: Published application
- Kind: A1
- Application number: 17/330,906
- Filed: 2021-05-26
- Published: 2021-09-09
- Earliest claimed: 2018-12-10
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: KOBA NATROSHVILI; KAY-ULRICH SCHOLL
- Web page: https://patentgrep.com/patent/US20210279894A1

## CPC

- G06V 10/82: using neural networks
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/25: Fusion techniques
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/251: {involving models}
- G06T 7/269: using gradient-based methods
- G06T 7/50: Depth or shape recovery
- G06T 7/579: from motion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A mechanism is described for facilitating depth and motion estimation in machine learning environments, according to one embodiment. A method of embodiments, as described herein, includes receiving a frame associated with a scene captured by one or more cameras of a computing device; processing the frame using a deep recurrent neural network architecture, wherein processing includes simultaneously predicating values associated with multiple loss functions corresponding to the frame; and estimating depth and motion based the predicted values.

## Claims

### Claim 1

At least one computer-readable medium comprising instructions which, when executed, cause a computing device to perform operations comprising:
receiving a frame associated with a scene;
predicating values associated with multiple loss functions corresponding to the frame; and
estimating one or more of depth or motion based on the predicted values.

### Claim 2 (depends on 1)

The computer-readable medium of claim 1, wherein the values are simultaneously predicted and comprise two or more of pixel depth, pixel velocity, pixel class and segmentation, or pixel optical flow.

### Claim 3 (depends on 1)

The computer-readable medium of claim 1, wherein the operations further comprise receiving and processing, by a neural network architecture, one or more of one or more previous frames for convolutional long short-term memory (LSTM) and odometry-based translation length, wherein the neural network architecture comprises a deep recurrent neural network architecture based on one or more deep recurrent neural networks or one or more convolutional LSTM layers.

### Claim 4 (depends on 1)

The computer-readable medium of claim 1, wherein the operations further comprise fusing together the multiple loss functions associated with the simultaneously predicted values in a cumulated cost function.

### Claim 5 (depends on 1)

The computer-readable medium of claim 1, wherein the operations further comprise estimating rotation matrixes and translation vectors for an object in the scene or one or more cameras capturing of the scene, wherein rotation matrixes and the translation vectors are fused together in a supervised form.

### Claim 6 (depends on 5)

The computer-readable medium of claim 5, wherein the operations further comprise constraining, based on odometry information, one or more of the rotation matrixes, translation vectors, and the simultaneously predicted values to estimate the depth of the scene and the motion of the one or more cameras.

### Claim 7 (depends on 1)

The computer-readable medium of claim 1, wherein the computing device comprises one or more processors comprising one or more of a graphics processor or an application processor, wherein the one or more processors are co-located on a common semiconductor package.

### Claim 8

A method comprising:
receiving a frame associated with a scene;
predicating values associated with multiple loss functions corresponding to the frame; and
estimating one or more of depth or motion based on the predicted values.

### Claim 9 (depends on 8)

The method of claim 8, wherein the values are simultaneously predicted and comprise two or more of pixel depth, pixel velocity, pixel class and segmentation, or pixel optical flow.

### Claim 10 (depends on 8)

The method of claim 8, further comprising receiving and processing, by a neural network architecture, one or more of one or more previous frames for convolutional long short-term memory (LSTM) and odometry-based translation length, wherein the neural network architecture comprises a deep recurrent neural network architecture based on one or more deep recurrent neural networks or one or more convolutional LSTM layers.

### Claim 11 (depends on 8)

The method of claim 8, further comprising fusing together the multiple loss functions associated with the simultaneously predicted values in a cumulated cost function.

### Claim 12 (depends on 8)

The method of claim 8, further comprising estimating rotation matrixes and translation vectors for an object in the scene or one or more cameras capturing of the scene, wherein rotation matrixes and the translation vectors are fused together in a supervised form.

### Claim 13 (depends on 12)

The method of claim 12, further comprising constraining, based on odometry information, one or more of the rotation matrixes, translation vectors, and the simultaneously predicted values to estimate the depth of the scene and the motion of the one or more cameras, wherein the one or more cameras are coupled to one or more processors comprising one or more of a graphics processor or an application processor, wherein the one or more processors are co-located on a common semiconductor package.

### Claim 14

An apparatus comprising:
one or more processors to:
receive a frame associated with a scene;
predicating values associated with multiple loss functions corresponding to the frame; and
estimate one or more of depth or motion based on the predicted values.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the values are simultaneously predicted and comprise two or more of pixel depth, pixel velocity, pixel class and segmentation, or pixel optical flow.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the one or more processors to receive and process, via a neural network architecture, one or more of one or more previous frames for convolutional long short-term memory (LSTM) and odometry-based translation length, wherein the neural network architecture comprises a deep recurrent neural network architecture based on one or more deep recurrent neural networks or one or more convolutional LSTM layers.

### Claim 17 (depends on 14)

The apparatus of claim 14, wherein the one or more processors to fuse together the multiple loss functions associated with the simultaneously predicted values in a cumulated cost function.

### Claim 18 (depends on 14)

The apparatus of claim 14, wherein the one or more processors to estimate rotation matrixes and translation vectors for an object in the scene or one or more cameras capturing of the scene, wherein rotation matrixes and the translation vectors are fused together in a supervised form.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the one or more processors to constrain, based on odometry information, one or more of the rotation matrixes, translation vectors, and the simultaneously predicted values to estimate the depth of the scene and the motion of the one or more cameras.

### Claim 20 (depends on 14)

The apparatus of claim 14, wherein the one or more processors comprising one or more of a graphics processor or an application processor, wherein the one or more processors are co-located on a common semiconductor package.